Sep 06 04:55:21.777309 ip-10-0-128-168 systemd[1]: Starting Kubernetes Kubelet...
Sep 06 04:55:22.213268 ip-10-0-128-168 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 04:55:22.213268 ip-10-0-128-168 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 06 04:55:22.213268 ip-10-0-128-168 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 04:55:22.213268 ip-10-0-128-168 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 06 04:55:22.213268 ip-10-0-128-168 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 04:55:22.215585 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.215477    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 06 04:55:22.219603 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219586    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 04:55:22.219603 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219601    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 04:55:22.219603 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219605    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219608    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219611    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219614    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219616    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219619    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219621    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219625    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219628    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219630    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219633    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219636    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219638    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219641    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219644    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219646    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219649    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219651    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219654    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219658    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 04:55:22.219709 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219661    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219664    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219667    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219669    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219672    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219674    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219677    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219679    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219682    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219684    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219687    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219690    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219693    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219696    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219698    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219701    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219704    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219707    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219709    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219712    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 04:55:22.220221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219715    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219718    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219720    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219723    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219726    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219728    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219731    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219733    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219736    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219738    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219741    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219743    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219746    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219749    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219752    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219755    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219757    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219760    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219764    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 04:55:22.220733 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219768    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219771    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219773    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219775    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219778    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219781    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219784    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219787    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219792    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219795    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219799    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219802    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219804    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219807    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219810    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219813    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219815    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219818    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219821    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 04:55:22.221221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219825    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219828    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219830    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219833    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219835    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219837    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219840    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219843    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219846    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219848    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219851    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219853    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219856    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219858    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219861    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219863    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219866    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219868    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.219871    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220567    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 04:55:22.221682 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220574    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220577    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220579    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220582    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220585    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220588    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220590    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220595    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220599    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220603    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220606    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220609    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220612    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220615    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220618    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220620    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220623    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220625    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220628    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 04:55:22.222208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220632    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220635    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220639    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220641    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220644    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220647    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220650    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220652    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220655    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220657    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220660    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220664    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220667    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220669    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220672    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220674    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220677    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220680    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220682    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220685    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 04:55:22.222705 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220687    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220690    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220692    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220695    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220697    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220699    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220702    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220704    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220706    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220709    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220711    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220714    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220717    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220719    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220722    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220725    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220727    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220730    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220733    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220736    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 04:55:22.223208 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220739    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220742    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220744    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220747    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220749    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220753    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220757    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220759    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220762    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220764    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220767    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220769    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220771    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220774    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220776    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220779    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220781    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220784    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220786    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220789    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 04:55:22.223688 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220791    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220794    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220796    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220799    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220801    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220804    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220806    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220809    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220812    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220814    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220817    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220820    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220823    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220825    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220828    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220830    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220833    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220835    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.220838    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221216    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 04:55:22.224188 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221221    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221224    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221227    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221230    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221233    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221235    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221238    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221241    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221243    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221246    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221248    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221250    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221253    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221255    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221258    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221260    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221263    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221266    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221268    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221272    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 04:55:22.224675 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221274    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221277    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221280    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221282    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221285    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221287    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221290    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221292    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221295    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221297    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221299    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221302    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221304    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221307    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221309    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221311    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221314    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221316    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221318    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221321    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 04:55:22.225183 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221323    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221327    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221331    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221334    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221336    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221339    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221341    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221344    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221347    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221351    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221354    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221357    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221360    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221363    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221367    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221370    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221373    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221375    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221378    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 04:55:22.225743 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221380    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221382    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221385    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221387    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221390    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221392    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221394    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221397    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221400    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221402    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221405    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221407    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221410    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221412    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221414    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221417    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221419    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221422    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221426    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 04:55:22.226347 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221429    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221433    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221437    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221439    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221442    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221444    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221447    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221450    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221453    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221455    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221457    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221460    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221462    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221465    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221467    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221470    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221472    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221474    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221477    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.221480    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221557    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 06 04:55:22.226802 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221600    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221607    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221612    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221616    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221619    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221623    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221627    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221630    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221633    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221636    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221639    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221642    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221646    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221649    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221652    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221655    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221658    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221665    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221668    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221671    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221674    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221677    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221681    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 06 04:55:22.227329 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221684    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221687    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221690    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221693    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221696    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221699    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221703    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221705    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221709    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221712    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221715    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221718    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221721    2580 flags.go:64] FLAG: --enable-server="true"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221724    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221727    2580 flags.go:64] FLAG: --event-burst="100"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221730    2580 flags.go:64] FLAG: --event-qps="50"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221733    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221736    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221739    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221742    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221745    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221749    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221752    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221755    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221758    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 06 04:55:22.227871 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221761    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221765    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221768    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221771    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221774    2580 flags.go:64] FLAG: --feature-gates=""
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221778    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221781    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221784    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221787    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221790    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221793    2580 flags.go:64] FLAG: --help="false"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221795    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221798    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221802    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221804    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221808    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221811    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221814    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221817    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221820    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221822    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221825    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221828    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221831    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 06 04:55:22.228551 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221834    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221836    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221839    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221842    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221846    2580 flags.go:64] FLAG: --lock-file=""
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221848    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221851    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221854    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221859    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221863    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221866    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221868    2580 flags.go:64] FLAG: --logging-format="text"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221871    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221875    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221877    2580 flags.go:64] FLAG: --manifest-url=""
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221881    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221885    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221888    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221892    2580 flags.go:64] FLAG: --max-pods="110"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221895    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221897    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221900    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221903    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221906    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221909    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 06 04:55:22.229172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221912    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221918    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221921    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221924    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221927    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221930    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221935    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221938    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221941    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221943    2580 flags.go:64] FLAG: --port="10250"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221946    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221950    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-007af1e21b3ebea47"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221954    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221956    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221959    2580 flags.go:64] FLAG: --register-node="true"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221962    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221965    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221969    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221972    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221975    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221978    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221982    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221985    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221988    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221990    2580 flags.go:64] FLAG: --runonce="false"
Sep 06 04:55:22.230144 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221993    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221996    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.221999    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222018    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222021    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222024    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222027    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222030    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222033    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222036    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222038    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222042    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222045    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222047    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222050    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222056    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222059    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222061    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222066    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222069    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222072    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222075    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222078    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222081    2580 flags.go:64] FLAG: --v="2"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222085    2580 flags.go:64] FLAG: --version="false"
Sep 06 04:55:22.230873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222089    2580 flags.go:64] FLAG: --vmodule=""
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222093    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.222096    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222228    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222233    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222236    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222239    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222242    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222245    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222248    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222251    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222254    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222256    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222259    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222262    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222264    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222267    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222270    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222272    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222275    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 04:55:22.231500 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222277    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222280    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222282    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222285    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222287    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222290    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222293    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222295    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222298    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222301    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222304    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222307    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222310    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222312    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222315    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222318    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222320    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222323    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222325    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222328    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 04:55:22.231983 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222330    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222333    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222335    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222338    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222340    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222343    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222345    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222347    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222350    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222352    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222355    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222357    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222360    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222362    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222364    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222367    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222369    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222372    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222375    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 04:55:22.232494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222379    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222381    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222384    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222387    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222391    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222395    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222398    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222401    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222403    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222406    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222408    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222411    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222413    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222416    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222418    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222421    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222424    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222426    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222428    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 04:55:22.232985 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222431    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222433    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222436    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222438    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222441    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222443    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222445    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222448    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222450    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222453    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222455    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222457    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222460    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222463    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222465    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222467    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222469    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222474    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222476    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222479    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 04:55:22.233470 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222481    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222484    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222486    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.222490    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.223193    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.231601    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.231719    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.232532    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.232540    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232599    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232604    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232607    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232611    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232614    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 04:55:22.233945 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232616    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232619    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232622    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232625    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232628    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232631    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232636    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232638    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232641    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232644    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232647    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232649    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232652    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232655    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232658    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232661    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232663    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232666    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232668    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 04:55:22.234308 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232671    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232673    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232676    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232678    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232681    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232684    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232687    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232690    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232693    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232695    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232697    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232700    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232703    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232705    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232707    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232710    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232713    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232715    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232718    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232721    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 04:55:22.234763 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232723    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232726    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232729    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232732    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232734    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232737    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232739    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232742    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232744    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232747    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232749    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232751    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232754    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232756    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232759    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232761    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232764    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232767    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232770    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232773    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 04:55:22.235263 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232776    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232778    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232780    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232784    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232787    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232790    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232792    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232795    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232797    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232800    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232802    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232805    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232807    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232809    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232812    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232815    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232818    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232820    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232823    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232825    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 04:55:22.235765 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232828    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232830    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232833    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232835    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232837    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232840    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232842    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232845    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232848    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232850    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232853    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232856    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232860    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232863    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232865    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 04:55:22.236378 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.232870    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232980    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232985    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232989    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232991    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232995    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.232998    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233018    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233023    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233026    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233028    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233032    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233037    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233042    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233044    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233047    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233050    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233052    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233055    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 04:55:22.236753 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233057    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233060    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233063    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233065    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233068    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233070    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233073    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233075    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233078    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233080    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233083    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233085    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233088    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233090    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233093    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233096    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233099    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233102    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233105    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233107    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 04:55:22.237221 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233110    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233113    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233115    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233117    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233120    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233122    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233124    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233127    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233130    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233132    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233134    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233137    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233139    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233142    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233144    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233146    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233149    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233152    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233154    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233157    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 04:55:22.237694 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233159    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233162    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233164    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233167    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233169    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233172    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233174    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233176    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233179    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233181    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233183    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233186    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233188    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233191    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233193    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233195    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233198    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233200    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233203    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233205    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 04:55:22.238184 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233209    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233211    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233214    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233216    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233218    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233221    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233223    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233226    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233228    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233230    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233233    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233235    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233238    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233241    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233243    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233246    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233249    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233252    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233255    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233257    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 04:55:22.238655 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:22.233260    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 04:55:22.239149 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.233265    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 04:55:22.239149 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.233367    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 06 04:55:22.239149 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.236861    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 06 04:55:22.239149 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.237864    2580 server.go:1019] "Starting client certificate rotation"
Sep 06 04:55:22.239149 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.237978    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 04:55:22.239149 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.238029    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 04:55:22.240885 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.240872    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 04:55:22.244505 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.244488    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 04:55:22.253272 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.253251    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 06 04:55:22.256434 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.256417    2580 log.go:25] "Validated CRI v1 image API"
Sep 06 04:55:22.257368 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.257351    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 06 04:55:22.261409 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.261390    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7c343aba-2c4a-455a-8c77-7e6e2f565246:/dev/nvme0n1p4 8a2b1b2c-f166-4408-bcbf-bd1b3c356560:/dev/nvme0n1p3]
Sep 06 04:55:22.261476 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.261409    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 06 04:55:22.269514 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.269387    2580 manager.go:216] Machine: {Timestamp:2026-09-06 04:55:22.268054582 +0000 UTC m=+0.364765286 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097930 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2329cd72875804cc56b2c791cfad06 SystemUUID:ec2329cd-7287-5804-cc56-b2c791cfad06 BootID:d703a2a4-5c2e-4336-903e-0f62f4931c68 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:8c:24:89:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:8c:24:89:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:7d:8b:f9:37:a4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 06 04:55:22.269514 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.269502    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 06 04:55:22.269638 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.269616    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 06 04:55:22.271042 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.271024    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 04:55:22.280632 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.273297    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 06 04:55:22.280773 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.280639    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-168.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 06 04:55:22.281498 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.281487    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 06 04:55:22.281530 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.281501    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 06 04:55:22.281530 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.281508    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 04:55:22.282317 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.282306    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 04:55:22.284030 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.284018    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 06 04:55:22.288233 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.288221    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 04:55:22.288355 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.288346    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 06 04:55:22.288389 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.288376    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 06 04:55:22.288389 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.288386    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 06 04:55:22.288441 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.288403    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 06 04:55:22.288441 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.288414    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 06 04:55:22.288441 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.288423    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 06 04:55:22.288808 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.288796    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 04:55:22.288848 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.288815    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 04:55:22.291318 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.291299    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 06 04:55:22.294032 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294018    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 06 04:55:22.294087 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294049    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 06 04:55:22.294872 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294859    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 06 04:55:22.294913 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294879    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 06 04:55:22.294913 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294885    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 06 04:55:22.294913 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294891    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 06 04:55:22.294913 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294900    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 06 04:55:22.294913 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294907    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 06 04:55:22.295055 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294921    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 06 04:55:22.295055 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294930    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 06 04:55:22.295055 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294936    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 06 04:55:22.295055 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294944    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 06 04:55:22.295055 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294959    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 06 04:55:22.295055 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.294964    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 06 04:55:22.296731 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.296717    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 06 04:55:22.296731 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.296727    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 06 04:55:22.299616 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.299564    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-168.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 06 04:55:22.299719 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.299624    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 06 04:55:22.299719 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.299625    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 06 04:55:22.300880 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.300865    2580 server.go:1268] "Started kubelet"
Sep 06 04:55:22.300981 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.300956    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 06 04:55:22.301059 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.300955    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 06 04:55:22.301104 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.301060    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 06 04:55:22.301610 ip-10-0-128-168 systemd[1]: Started Kubernetes Kubelet.
Sep 06 04:55:22.304212 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.304195    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 06 04:55:22.305120 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.305104    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 06 04:55:22.305888 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.305871    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 06 04:55:22.309662 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.309649    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 06 04:55:22.310314 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.310296    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 06 04:55:22.310453 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.310431    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:22.311150 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311131    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 06 04:55:22.311150 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311146    2580 factory.go:55] Registering systemd factory
Sep 06 04:55:22.311280 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311161    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 06 04:55:22.311280 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311129    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 06 04:55:22.311280 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311185    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 06 04:55:22.311433 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311343    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 06 04:55:22.311433 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311432    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 06 04:55:22.311592 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311444    2580 factory.go:153] Registering CRI-O factory
Sep 06 04:55:22.311592 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311458    2580 factory.go:223] Registration of the crio container factory successfully
Sep 06 04:55:22.311592 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311500    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 06 04:55:22.311592 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311520    2580 factory.go:103] Registering Raw factory
Sep 06 04:55:22.311592 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.311533    2580 manager.go:1196] Started watching for new ooms in manager
Sep 06 04:55:22.314030 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.312047    2580 manager.go:349] Starting recovery of all containers
Sep 06 04:55:22.317245 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.317206    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x2tm8"
Sep 06 04:55:22.318033 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.317873    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 06 04:55:22.318270 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.318109    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 06 04:55:22.322911 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.318257    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-168.ec2.internal.18d2a25cfdf4358f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-168.ec2.internal,UID:ip-10-0-128-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-168.ec2.internal,},FirstTimestamp:2026-09-06 04:55:22.300839311 +0000 UTC m=+0.397549996,LastTimestamp:2026-09-06 04:55:22.300839311 +0000 UTC m=+0.397549996,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-168.ec2.internal,}"
Sep 06 04:55:22.325507 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.325481    2580 manager.go:354] Recovery completed
Sep 06 04:55:22.328312 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.328294    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x2tm8"
Sep 06 04:55:22.329620 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.329606    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 04:55:22.331290 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.331274    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 04:55:22.331357 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.331310    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 04:55:22.331357 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.331326    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Sep 06 04:55:22.331856 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.331843    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 06 04:55:22.331856 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.331849    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 04:55:22.331856 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.331854    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 06 04:55:22.332030 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.331867    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 04:55:22.332492 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.332477    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 04:55:22.332560 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.332504    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 04:55:22.332560 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.332536    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Sep 06 04:55:22.333538 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.333465    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-168.ec2.internal.18d2a25cffc4db18  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-168.ec2.internal,UID:ip-10-0-128-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-168.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-168.ec2.internal,},FirstTimestamp:2026-09-06 04:55:22.331290392 +0000 UTC m=+0.428001084,LastTimestamp:2026-09-06 04:55:22.331290392 +0000 UTC m=+0.428001084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-168.ec2.internal,}"
Sep 06 04:55:22.333946 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.333935    2580 policy_none.go:49] "None policy: Start"
Sep 06 04:55:22.333983 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.333951    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 06 04:55:22.333983 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.333961    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 06 04:55:22.335995 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.335986    2580 policy_none.go:47] "Start"
Sep 06 04:55:22.367631 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.367617    2580 manager.go:337] "Starting Device Plugin manager"
Sep 06 04:55:22.367724 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.367663    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 06 04:55:22.367724 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.367676    2580 server.go:87] "Starting device plugin registration server"
Sep 06 04:55:22.367902 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.367888    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 06 04:55:22.367963 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.367904    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 06 04:55:22.368192 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.368057    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 06 04:55:22.368192 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.368127    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 06 04:55:22.368192 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.368135    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 06 04:55:22.368732 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.368714    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 06 04:55:22.368808 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.368766    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:22.384277 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.384244    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 06 04:55:22.396729 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.385465    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 06 04:55:22.396729 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.385504    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 06 04:55:22.396729 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.385525    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 06 04:55:22.396729 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.385562    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 06 04:55:22.396729 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.387363    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 04:55:22.468915 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.468847    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 04:55:22.469594 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.469578    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 04:55:22.469645 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.469605    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 04:55:22.469645 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.469619    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Sep 06 04:55:22.469645 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.469640    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.478209 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.478197    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.478255 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.478216    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-168.ec2.internal\": node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:22.485843 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.485828    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"]
Sep 06 04:55:22.485888 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.485884    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 04:55:22.487297 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.487283    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 04:55:22.487352 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.487309    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 04:55:22.487352 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.487321    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Sep 06 04:55:22.489335 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.489324    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 04:55:22.489443 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.489431    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.489503 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.489457    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 04:55:22.489956 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.489929    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 04:55:22.489956 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.489947    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 04:55:22.489956 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.489956    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 04:55:22.490129 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.489968    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Sep 06 04:55:22.490129 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.489970    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 04:55:22.490129 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.489986    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Sep 06 04:55:22.492508 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.492493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.492578 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.492517    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 04:55:22.493171 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.493153    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 04:55:22.493247 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.493179    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 04:55:22.493247 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.493203    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Sep 06 04:55:22.504688 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.504673    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:22.524609 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.524592    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-168.ec2.internal\" not found" node="ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.528948 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.528931    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-168.ec2.internal\" not found" node="ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.605698 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.605681    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:22.611946 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.611931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/536e7fcc5b50e39897826a8d67dc9141-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"536e7fcc5b50e39897826a8d67dc9141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.612027 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.611952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a6422a356691d2cbb645b3d74be5d53-config\") pod \"kube-apiserver-proxy-ip-10-0-128-168.ec2.internal\" (UID: \"2a6422a356691d2cbb645b3d74be5d53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.612027 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.611973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/536e7fcc5b50e39897826a8d67dc9141-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"536e7fcc5b50e39897826a8d67dc9141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.706156 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.706114    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:22.712398 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.712377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/536e7fcc5b50e39897826a8d67dc9141-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"536e7fcc5b50e39897826a8d67dc9141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.712450 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.712406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/536e7fcc5b50e39897826a8d67dc9141-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"536e7fcc5b50e39897826a8d67dc9141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.712450 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.712423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a6422a356691d2cbb645b3d74be5d53-config\") pod \"kube-apiserver-proxy-ip-10-0-128-168.ec2.internal\" (UID: \"2a6422a356691d2cbb645b3d74be5d53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.712512 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.712469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a6422a356691d2cbb645b3d74be5d53-config\") pod \"kube-apiserver-proxy-ip-10-0-128-168.ec2.internal\" (UID: \"2a6422a356691d2cbb645b3d74be5d53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.712512 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.712476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/536e7fcc5b50e39897826a8d67dc9141-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"536e7fcc5b50e39897826a8d67dc9141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.712575 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.712469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/536e7fcc5b50e39897826a8d67dc9141-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"536e7fcc5b50e39897826a8d67dc9141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.806808 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.806765    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:22.828208 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.828190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.833054 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:22.833035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:22.907186 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:22.907146    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:23.007640 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:23.007619    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:23.108199 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:23.108142    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:23.208617 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:23.208595    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:23.238094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.238077    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 06 04:55:23.238415 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.238190    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 04:55:23.309027 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:23.308996    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Sep 06 04:55:23.314477 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.314462    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 04:55:23.331452 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.331427    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 04:50:24 +0000 UTC" deadline="2028-06-06 00:35:46.04865369 +0000 UTC"
Sep 06 04:55:23.331452 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.331450    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15331h40m22.71720645s"
Sep 06 04:55:23.410518 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.410475    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:23.424035 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.424019    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 04:55:23.426149 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.426135    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Sep 06 04:55:23.435072 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.435058    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 04:55:23.488659 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.488635    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 04:55:23.574863 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:23.574694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536e7fcc5b50e39897826a8d67dc9141.slice/crio-df1a36806c4f4350c8945c6df74913f4af094f133fc0e9292d79d0a693b72b51 WatchSource:0}: Error finding container df1a36806c4f4350c8945c6df74913f4af094f133fc0e9292d79d0a693b72b51: Status 404 returned error can't find the container with id df1a36806c4f4350c8945c6df74913f4af094f133fc0e9292d79d0a693b72b51
Sep 06 04:55:23.580609 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.580595    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 04:55:23.731907 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:23.731859    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 04:55:26.245576 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.245549    2580 apiserver.go:52] "Watching apiserver"
Sep 06 04:55:26.252546 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.252522    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 06 04:55:26.254890 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.254859    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-csrcb","kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2","openshift-dns/node-resolver-kwqp6","openshift-image-registry/node-ca-hr7nb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal","openshift-network-diagnostics/network-check-target-cqc98","openshift-network-operator/iptables-alerter-42d2r","openshift-ovn-kubernetes/ovnkube-node-l6gsk","kube-system/global-pull-secret-syncer-g6f2s","kube-system/konnectivity-agent-d24f8","openshift-cluster-node-tuning-operator/tuned-vgwr4","openshift-multus/multus-additional-cni-plugins-4568p","openshift-multus/multus-vrnfr"]
Sep 06 04:55:26.257328 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.257299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:26.257432 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.257378    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:26.259823 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.259070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.262519 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.262495    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 06 04:55:26.262625 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.262606    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 06 04:55:26.262767 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.262750    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-425w4\""
Sep 06 04:55:26.262817 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.262785    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 06 04:55:26.262851 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.262832    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 06 04:55:26.262982 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.262971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.263085 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.262997    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 06 04:55:26.264810 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.264319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.264916 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.264840    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 06 04:55:26.265811 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.265795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 06 04:55:26.266000 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.265984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7b59q\""
Sep 06 04:55:26.266203 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.266186    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 06 04:55:26.266578 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.266561    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 06 04:55:26.266805 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.266788    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xtxlb\""
Sep 06 04:55:26.267002 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.266984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 06 04:55:26.267217 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.267200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.267302 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.267264    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 06 04:55:26.269006 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.268987    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.269634 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.269616    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 06 04:55:26.269833 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.269818    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 06 04:55:26.269984 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.269963    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 06 04:55:26.270056 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.269997    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxvk4\""
Sep 06 04:55:26.271677 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.271246    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 06 04:55:26.271677 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.271378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-chmnx\""
Sep 06 04:55:26.271677 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.271546    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.273664 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.273646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:26.273754 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.273721    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:26.274093 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.274071    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 06 04:55:26.274245 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.274149    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 06 04:55:26.275281 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.274073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 06 04:55:26.275541 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.275520    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7sjxc\""
Sep 06 04:55:26.275818 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.275783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-run-multus-certs\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.275818 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.275833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a99b800d-76a9-4349-82f8-55804daefcde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.276080 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.275868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.276080 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.275898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-registration-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.276080 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-run-netns\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.276486 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-hostroot\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.276486 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-conf-dir\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.276486 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a99b800d-76a9-4349-82f8-55804daefcde-cnibin\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.276486 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhpk9\" (UniqueName: \"kubernetes.io/projected/a99b800d-76a9-4349-82f8-55804daefcde-kube-api-access-dhpk9\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.276486 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-socket-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.276486 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-etc-selinux\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.276486 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-socket-dir-parent\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.277218 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53aa3ac4-3e77-40df-9136-ff0e41c2606b-serviceca\") pod \"node-ca-hr7nb\" (UID: \"53aa3ac4-3e77-40df-9136-ff0e41c2606b\") " pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.277218 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-etc-kubernetes\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.277218 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78jg8\" (UniqueName: \"kubernetes.io/projected/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-kube-api-access-78jg8\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.277218 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-device-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.277218 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-cnibin\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.277218 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a99b800d-76a9-4349-82f8-55804daefcde-system-cni-dir\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.277218 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.276993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a99b800d-76a9-4349-82f8-55804daefcde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.277218 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277012    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 06 04:55:26.277218 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6g8\" (UniqueName: \"kubernetes.io/projected/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-kube-api-access-zd6g8\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.277218 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbq6\" (UniqueName: \"kubernetes.io/projected/53aa3ac4-3e77-40df-9136-ff0e41c2606b-kube-api-access-kmbq6\") pod \"node-ca-hr7nb\" (UID: \"53aa3ac4-3e77-40df-9136-ff0e41c2606b\") " pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-cni-binary-copy\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53aa3ac4-3e77-40df-9136-ff0e41c2606b-host\") pod \"node-ca-hr7nb\" (UID: \"53aa3ac4-3e77-40df-9136-ff0e41c2606b\") " pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-var-lib-cni-bin\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-var-lib-cni-multus\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-daemon-config\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a99b800d-76a9-4349-82f8-55804daefcde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277597    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-var-lib-kubelet\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-sys-fs\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq7wg\" (UniqueName: \"kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg\") pod \"network-check-target-cqc98\" (UID: \"6ce9940b-e5b0-4647-bf32-532f1806d6a3\") " pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-system-cni-dir\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96-tmp-dir\") pod \"node-resolver-kwqp6\" (UID: \"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96\") " pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a99b800d-76a9-4349-82f8-55804daefcde-cni-binary-copy\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.277978 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.277892    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 06 04:55:26.278689 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.278256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-run-k8s-cni-cncf-io\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.278689 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.278283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-cni-dir\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.278689 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.278333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96-hosts-file\") pod \"node-resolver-kwqp6\" (UID: \"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96\") " pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.278689 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.278366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vq6p\" (UniqueName: \"kubernetes.io/projected/fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96-kube-api-access-8vq6p\") pod \"node-resolver-kwqp6\" (UID: \"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96\") " pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.278689 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.278459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.279572 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.279545    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 04:55:26.281553 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.280872    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 06 04:55:26.281553 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.281106    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jmthr\""
Sep 06 04:55:26.281553 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.281266    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 06 04:55:26.281553 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.281341    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 06 04:55:26.282388 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.282212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:26.282388 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.282275    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:26.284477 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.284457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:26.287183 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.286896    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 06 04:55:26.287183 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.286927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.287183 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.286951    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5xbtf\""
Sep 06 04:55:26.287183 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.287115    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 06 04:55:26.289382 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.288943    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 06 04:55:26.289382 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.289246    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4tnz5\""
Sep 06 04:55:26.289873 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.289727    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 06 04:55:26.347561 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.347518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal" event={"ID":"2a6422a356691d2cbb645b3d74be5d53","Type":"ContainerStarted","Data":"86c93ad3185f97e866635c7c89af2e9f8bb9d72f37b3a63ed81d9260966c88e8"}
Sep 06 04:55:26.349042 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.349018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal" event={"ID":"536e7fcc5b50e39897826a8d67dc9141","Type":"ContainerStarted","Data":"df1a36806c4f4350c8945c6df74913f4af094f133fc0e9292d79d0a693b72b51"}
Sep 06 04:55:26.367575 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.367511    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 06 04:55:26.369062 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.369003    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pr6fq"
Sep 06 04:55:26.375707 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.375686    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pr6fq"
Sep 06 04:55:26.378597 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-sysctl-d\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.378690 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-sys\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.378690 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m82cd\" (UniqueName: \"kubernetes.io/projected/93f31079-87fb-497d-bdc4-5ae1b391f5ad-kube-api-access-m82cd\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.378690 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-slash\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.378690 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7da02b5e-ca23-4376-9c84-7b1ff2713408-ovnkube-script-lib\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.378872 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w7jv\" (UniqueName: \"kubernetes.io/projected/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-kube-api-access-6w7jv\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:26.378872 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-sys-fs\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.378872 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-etc-openvswitch\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.378872 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-system-cni-dir\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.378872 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-sys-fs\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.378872 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-tuned\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-sysconfig\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-systemd-units\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-run-netns\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7da02b5e-ca23-4376-9c84-7b1ff2713408-ovnkube-config\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-modprobe-d\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-system-cni-dir\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.378955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-run\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526tf\" (UniqueName: \"kubernetes.io/projected/7da02b5e-ca23-4376-9c84-7b1ff2713408-kube-api-access-526tf\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bb6a418-6784-4b50-bfc4-f8331eb404f1-iptables-alerter-script\") pod \"iptables-alerter-42d2r\" (UID: \"4bb6a418-6784-4b50-bfc4-f8331eb404f1\") " pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.379064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2153f522-a60a-4c12-8aea-bf25f12e162e-agent-certs\") pod \"konnectivity-agent-d24f8\" (UID: \"2153f522-a60a-4c12-8aea-bf25f12e162e\") " pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:26.379400 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.379400 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-run-netns\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.379400 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-conf-dir\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.379400 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-run-netns\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.379400 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-socket-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.379400 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-conf-dir\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.379400 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-etc-selinux\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.379400 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d08567e7-7463-408b-9d6f-d6edca57b781-kubelet-config\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:26.379698 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-run-openvswitch\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.379698 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7da02b5e-ca23-4376-9c84-7b1ff2713408-env-overrides\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.379698 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-etc-selinux\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.379698 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-kubernetes\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.379698 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-sysctl-conf\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.379698 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53aa3ac4-3e77-40df-9136-ff0e41c2606b-serviceca\") pod \"node-ca-hr7nb\" (UID: \"53aa3ac4-3e77-40df-9136-ff0e41c2606b\") " pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.379698 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7da02b5e-ca23-4376-9c84-7b1ff2713408-ovn-node-metrics-cert\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.379698 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:26.379698 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a99b800d-76a9-4349-82f8-55804daefcde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.379698 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2153f522-a60a-4c12-8aea-bf25f12e162e-konnectivity-ca\") pod \"konnectivity-agent-d24f8\" (UID: \"2153f522-a60a-4c12-8aea-bf25f12e162e\") " pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-systemd\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbq6\" (UniqueName: \"kubernetes.io/projected/53aa3ac4-3e77-40df-9136-ff0e41c2606b-kube-api-access-kmbq6\") pod \"node-ca-hr7nb\" (UID: \"53aa3ac4-3e77-40df-9136-ff0e41c2606b\") " pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-cni-binary-copy\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53aa3ac4-3e77-40df-9136-ff0e41c2606b-host\") pod \"node-ca-hr7nb\" (UID: \"53aa3ac4-3e77-40df-9136-ff0e41c2606b\") " pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-var-lib-cni-multus\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-daemon-config\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bb6a418-6784-4b50-bfc4-f8331eb404f1-host-slash\") pod \"iptables-alerter-42d2r\" (UID: \"4bb6a418-6784-4b50-bfc4-f8331eb404f1\") " pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-var-lib-kubelet\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7wg\" (UniqueName: \"kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg\") pod \"network-check-target-cqc98\" (UID: \"6ce9940b-e5b0-4647-bf32-532f1806d6a3\") " pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96-tmp-dir\") pod \"node-resolver-kwqp6\" (UID: \"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96\") " pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a99b800d-76a9-4349-82f8-55804daefcde-cni-binary-copy\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f31079-87fb-497d-bdc4-5ae1b391f5ad-tmp\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.380007 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.379989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-kubelet\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-cni-netd\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-socket-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-run-k8s-cni-cncf-io\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-var-lib-kubelet\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-host\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-node-log\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-run-ovn-kubernetes\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-cni-dir\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96-hosts-file\") pod \"node-resolver-kwqp6\" (UID: \"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96\") " pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vq6p\" (UniqueName: \"kubernetes.io/projected/fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96-kube-api-access-8vq6p\") pod \"node-resolver-kwqp6\" (UID: \"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96\") " pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-run-multus-certs\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a99b800d-76a9-4349-82f8-55804daefcde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-registration-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-cni-bin\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktv47\" (UniqueName: \"kubernetes.io/projected/4bb6a418-6784-4b50-bfc4-f8331eb404f1-kube-api-access-ktv47\") pod \"iptables-alerter-42d2r\" (UID: \"4bb6a418-6784-4b50-bfc4-f8331eb404f1\") " pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.380533 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-hostroot\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a99b800d-76a9-4349-82f8-55804daefcde-cnibin\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhpk9\" (UniqueName: \"kubernetes.io/projected/a99b800d-76a9-4349-82f8-55804daefcde-kube-api-access-dhpk9\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53aa3ac4-3e77-40df-9136-ff0e41c2606b-serviceca\") pod \"node-ca-hr7nb\" (UID: \"53aa3ac4-3e77-40df-9136-ff0e41c2606b\") " pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-run-systemd\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-var-lib-openvswitch\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-log-socket\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-lib-modules\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-var-lib-kubelet\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-socket-dir-parent\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-etc-kubernetes\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78jg8\" (UniqueName: \"kubernetes.io/projected/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-kube-api-access-78jg8\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-device-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d08567e7-7463-408b-9d6f-d6edca57b781-dbus\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-run-ovn\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-cnibin\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a99b800d-76a9-4349-82f8-55804daefcde-system-cni-dir\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.381594 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6g8\" (UniqueName: \"kubernetes.io/projected/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-kube-api-access-zd6g8\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.381594 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.380979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-var-lib-cni-bin\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381594 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-var-lib-cni-bin\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381594 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-daemon-config\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381594 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53aa3ac4-3e77-40df-9136-ff0e41c2606b-host\") pod \"node-ca-hr7nb\" (UID: \"53aa3ac4-3e77-40df-9136-ff0e41c2606b\") " pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.381594 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-var-lib-cni-multus\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381594 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-cni-binary-copy\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381594 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96-tmp-dir\") pod \"node-resolver-kwqp6\" (UID: \"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96\") " pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.381594 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-run-k8s-cni-cncf-io\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381861 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-host-run-multus-certs\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381861 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-socket-dir-parent\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381861 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-hostroot\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381861 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a99b800d-76a9-4349-82f8-55804daefcde-cnibin\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.381861 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-etc-kubernetes\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381861 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a99b800d-76a9-4349-82f8-55804daefcde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.381861 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-registration-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.381861 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-multus-cni-dir\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.381861 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96-hosts-file\") pod \"node-resolver-kwqp6\" (UID: \"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96\") " pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.382221 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a99b800d-76a9-4349-82f8-55804daefcde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.382221 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.381951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-device-dir\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.391543 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.389615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a99b800d-76a9-4349-82f8-55804daefcde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.391543 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.389670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-cnibin\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.391543 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.389677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a99b800d-76a9-4349-82f8-55804daefcde-cni-binary-copy\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.391543 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.389791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a99b800d-76a9-4349-82f8-55804daefcde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.391543 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.390446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a99b800d-76a9-4349-82f8-55804daefcde-system-cni-dir\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.391543 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.390739    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 04:55:26.391543 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.390760    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 04:55:26.391543 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.390775    2580 projected.go:196] Error preparing data for projected volume kube-api-access-zq7wg for pod openshift-network-diagnostics/network-check-target-cqc98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:26.391543 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.390849    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg podName:6ce9940b-e5b0-4647-bf32-532f1806d6a3 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:26.890820425 +0000 UTC m=+3.032380348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zq7wg" (UniqueName: "kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg") pod "network-check-target-cqc98" (UID: "6ce9940b-e5b0-4647-bf32-532f1806d6a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:26.391543 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.390849    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 06 04:55:26.396704 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.396679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78jg8\" (UniqueName: \"kubernetes.io/projected/ebf6fbec-f0bf-44ba-99dd-355169f9da5e-kube-api-access-78jg8\") pod \"multus-vrnfr\" (UID: \"ebf6fbec-f0bf-44ba-99dd-355169f9da5e\") " pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.397408 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.397384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vq6p\" (UniqueName: \"kubernetes.io/projected/fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96-kube-api-access-8vq6p\") pod \"node-resolver-kwqp6\" (UID: \"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96\") " pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.398041 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.398009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbq6\" (UniqueName: \"kubernetes.io/projected/53aa3ac4-3e77-40df-9136-ff0e41c2606b-kube-api-access-kmbq6\") pod \"node-ca-hr7nb\" (UID: \"53aa3ac4-3e77-40df-9136-ff0e41c2606b\") " pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.403255 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.402435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhpk9\" (UniqueName: \"kubernetes.io/projected/a99b800d-76a9-4349-82f8-55804daefcde-kube-api-access-dhpk9\") pod \"multus-additional-cni-plugins-4568p\" (UID: \"a99b800d-76a9-4349-82f8-55804daefcde\") " pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.406569 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.406547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6g8\" (UniqueName: \"kubernetes.io/projected/7d8c10d8-49ae-4bb5-8083-6c2596d4321a-kube-api-access-zd6g8\") pod \"aws-ebs-csi-driver-node-79zq2\" (UID: \"7d8c10d8-49ae-4bb5-8083-6c2596d4321a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.492697 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d08567e7-7463-408b-9d6f-d6edca57b781-dbus\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:26.492835 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-run-ovn\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.492835 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-sysctl-d\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.492835 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-sys\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.492835 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m82cd\" (UniqueName: \"kubernetes.io/projected/93f31079-87fb-497d-bdc4-5ae1b391f5ad-kube-api-access-m82cd\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.492835 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-slash\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.492835 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7da02b5e-ca23-4376-9c84-7b1ff2713408-ovnkube-script-lib\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w7jv\" (UniqueName: \"kubernetes.io/projected/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-kube-api-access-6w7jv\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-etc-openvswitch\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-tuned\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-sysconfig\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-systemd-units\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-run-netns\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7da02b5e-ca23-4376-9c84-7b1ff2713408-ovnkube-config\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-modprobe-d\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-run\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-sysctl-d\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.492983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526tf\" (UniqueName: \"kubernetes.io/projected/7da02b5e-ca23-4376-9c84-7b1ff2713408-kube-api-access-526tf\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bb6a418-6784-4b50-bfc4-f8331eb404f1-iptables-alerter-script\") pod \"iptables-alerter-42d2r\" (UID: \"4bb6a418-6784-4b50-bfc4-f8331eb404f1\") " pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2153f522-a60a-4c12-8aea-bf25f12e162e-agent-certs\") pod \"konnectivity-agent-d24f8\" (UID: \"2153f522-a60a-4c12-8aea-bf25f12e162e\") " pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d08567e7-7463-408b-9d6f-d6edca57b781-kubelet-config\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-run-openvswitch\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493094 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d08567e7-7463-408b-9d6f-d6edca57b781-dbus\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7da02b5e-ca23-4376-9c84-7b1ff2713408-env-overrides\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-kubernetes\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-sysctl-conf\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7da02b5e-ca23-4376-9c84-7b1ff2713408-ovn-node-metrics-cert\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2153f522-a60a-4c12-8aea-bf25f12e162e-konnectivity-ca\") pod \"konnectivity-agent-d24f8\" (UID: \"2153f522-a60a-4c12-8aea-bf25f12e162e\") " pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-slash\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-systemd\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bb6a418-6784-4b50-bfc4-f8331eb404f1-host-slash\") pod \"iptables-alerter-42d2r\" (UID: \"4bb6a418-6784-4b50-bfc4-f8331eb404f1\") " pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-systemd\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-run\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493652 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-modprobe-d\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.493986 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-run-ovn\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493986 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7da02b5e-ca23-4376-9c84-7b1ff2713408-ovnkube-script-lib\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.493986 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.493229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-sys\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.494088 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7da02b5e-ca23-4376-9c84-7b1ff2713408-ovnkube-config\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.494516 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bb6a418-6784-4b50-bfc4-f8331eb404f1-host-slash\") pod \"iptables-alerter-42d2r\" (UID: \"4bb6a418-6784-4b50-bfc4-f8331eb404f1\") " pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.494516 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-systemd-units\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.494516 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-run-netns\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.494516 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-sysconfig\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.494516 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d08567e7-7463-408b-9d6f-d6edca57b781-kubelet-config\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:26.494516 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-sysctl-conf\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.494843 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-kubernetes\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.494843 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bb6a418-6784-4b50-bfc4-f8331eb404f1-iptables-alerter-script\") pod \"iptables-alerter-42d2r\" (UID: \"4bb6a418-6784-4b50-bfc4-f8331eb404f1\") " pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.494843 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7da02b5e-ca23-4376-9c84-7b1ff2713408-env-overrides\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.494843 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-run-openvswitch\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.494843 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.494627    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:26.494843 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.494694    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs podName:d3a5a3f7-dc34-4f63-9fa9-5660515f8000 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:26.994675948 +0000 UTC m=+3.136235868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs") pod "network-metrics-daemon-csrcb" (UID: "d3a5a3f7-dc34-4f63-9fa9-5660515f8000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:26.494843 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-etc-openvswitch\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.494843 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f31079-87fb-497d-bdc4-5ae1b391f5ad-tmp\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.494843 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-kubelet\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-cni-netd\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-kubelet\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-var-lib-kubelet\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-host\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-cni-netd\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-node-log\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-run-ovn-kubernetes\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.494986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-var-lib-kubelet\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-host\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-cni-bin\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2153f522-a60a-4c12-8aea-bf25f12e162e-konnectivity-ca\") pod \"konnectivity-agent-d24f8\" (UID: \"2153f522-a60a-4c12-8aea-bf25f12e162e\") " pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-node-log\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktv47\" (UniqueName: \"kubernetes.io/projected/4bb6a418-6784-4b50-bfc4-f8331eb404f1-kube-api-access-ktv47\") pod \"iptables-alerter-42d2r\" (UID: \"4bb6a418-6784-4b50-bfc4-f8331eb404f1\") " pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-cni-bin\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.495348 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-host-run-ovn-kubernetes\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-run-systemd\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-var-lib-openvswitch\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-log-socket\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-lib-modules\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.495184    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-var-lib-openvswitch\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-log-socket\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.495272    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret podName:d08567e7-7463-408b-9d6f-d6edca57b781 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:26.995255647 +0000 UTC m=+3.136815573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret") pod "global-pull-secret-syncer-g6f2s" (UID: "d08567e7-7463-408b-9d6f-d6edca57b781") : object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7da02b5e-ca23-4376-9c84-7b1ff2713408-run-systemd\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.496364 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.495517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93f31079-87fb-497d-bdc4-5ae1b391f5ad-lib-modules\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.496896 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.496882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f31079-87fb-497d-bdc4-5ae1b391f5ad-tmp\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.497403 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.497351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93f31079-87fb-497d-bdc4-5ae1b391f5ad-etc-tuned\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.497509 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.497490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2153f522-a60a-4c12-8aea-bf25f12e162e-agent-certs\") pod \"konnectivity-agent-d24f8\" (UID: \"2153f522-a60a-4c12-8aea-bf25f12e162e\") " pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:26.497648 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.497606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7da02b5e-ca23-4376-9c84-7b1ff2713408-ovn-node-metrics-cert\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.501148 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.501121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m82cd\" (UniqueName: \"kubernetes.io/projected/93f31079-87fb-497d-bdc4-5ae1b391f5ad-kube-api-access-m82cd\") pod \"tuned-vgwr4\" (UID: \"93f31079-87fb-497d-bdc4-5ae1b391f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.502285 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.502244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-526tf\" (UniqueName: \"kubernetes.io/projected/7da02b5e-ca23-4376-9c84-7b1ff2713408-kube-api-access-526tf\") pod \"ovnkube-node-l6gsk\" (UID: \"7da02b5e-ca23-4376-9c84-7b1ff2713408\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.503133 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.503108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktv47\" (UniqueName: \"kubernetes.io/projected/4bb6a418-6784-4b50-bfc4-f8331eb404f1-kube-api-access-ktv47\") pod \"iptables-alerter-42d2r\" (UID: \"4bb6a418-6784-4b50-bfc4-f8331eb404f1\") " pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.503920 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.503897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w7jv\" (UniqueName: \"kubernetes.io/projected/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-kube-api-access-6w7jv\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:26.576285 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.576255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4568p"
Sep 06 04:55:26.587069 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.587047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2"
Sep 06 04:55:26.587216 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:26.587188    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99b800d_76a9_4349_82f8_55804daefcde.slice/crio-de67dbf9e6e78f29575cd0d6d9a35b62676264a33ff367ef521726795a04087d WatchSource:0}: Error finding container de67dbf9e6e78f29575cd0d6d9a35b62676264a33ff367ef521726795a04087d: Status 404 returned error can't find the container with id de67dbf9e6e78f29575cd0d6d9a35b62676264a33ff367ef521726795a04087d
Sep 06 04:55:26.601756 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.601738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kwqp6"
Sep 06 04:55:26.606928 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.606906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hr7nb"
Sep 06 04:55:26.617651 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.617527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vrnfr"
Sep 06 04:55:26.624304 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.624147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:26.633898 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.633873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-42d2r"
Sep 06 04:55:26.641544 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.641419    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:26.649023 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.648997    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vgwr4"
Sep 06 04:55:26.708397 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.708375    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 04:55:26.898029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.897946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7wg\" (UniqueName: \"kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg\") pod \"network-check-target-cqc98\" (UID: \"6ce9940b-e5b0-4647-bf32-532f1806d6a3\") " pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:26.898189 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.898130    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 04:55:26.898189 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.898171    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 04:55:26.898189 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.898185    2580 projected.go:196] Error preparing data for projected volume kube-api-access-zq7wg for pod openshift-network-diagnostics/network-check-target-cqc98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:26.898313 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.898249    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg podName:6ce9940b-e5b0-4647-bf32-532f1806d6a3 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:27.898230628 +0000 UTC m=+4.039790562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zq7wg" (UniqueName: "kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg") pod "network-check-target-cqc98" (UID: "6ce9940b-e5b0-4647-bf32-532f1806d6a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:26.998633 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.998599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:26.998771 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:26.998675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:26.998771 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.998735    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:26.998852 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.998780    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:26.998852 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.998818    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret podName:d08567e7-7463-408b-9d6f-d6edca57b781 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:27.998795085 +0000 UTC m=+4.140355028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret") pod "global-pull-secret-syncer-g6f2s" (UID: "d08567e7-7463-408b-9d6f-d6edca57b781") : object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:26.998952 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:26.998854    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs podName:d3a5a3f7-dc34-4f63-9fa9-5660515f8000 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:27.998843275 +0000 UTC m=+4.140403196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs") pod "network-metrics-daemon-csrcb" (UID: "d3a5a3f7-dc34-4f63-9fa9-5660515f8000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:27.352261 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:27.352223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4568p" event={"ID":"a99b800d-76a9-4349-82f8-55804daefcde","Type":"ContainerStarted","Data":"de67dbf9e6e78f29575cd0d6d9a35b62676264a33ff367ef521726795a04087d"}
Sep 06 04:55:27.379564 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:27.379529    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 04:50:26 +0000 UTC" deadline="2028-04-04 20:32:56.00723015 +0000 UTC"
Sep 06 04:55:27.379564 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:27.379561    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13839h37m28.627673411s"
Sep 06 04:55:27.642874 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:27.642852    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8c10d8_49ae_4bb5_8083_6c2596d4321a.slice/crio-6c13569ab2480b0246ecf1a7feb3d3b6650251b1ef784b81c46e747864f1a75b WatchSource:0}: Error finding container 6c13569ab2480b0246ecf1a7feb3d3b6650251b1ef784b81c46e747864f1a75b: Status 404 returned error can't find the container with id 6c13569ab2480b0246ecf1a7feb3d3b6650251b1ef784b81c46e747864f1a75b
Sep 06 04:55:27.643584 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:27.643558    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f31079_87fb_497d_bdc4_5ae1b391f5ad.slice/crio-7b1d15b5824ff965dbae0db96da7e1fb888a030605c99b050d2408114312b3b1 WatchSource:0}: Error finding container 7b1d15b5824ff965dbae0db96da7e1fb888a030605c99b050d2408114312b3b1: Status 404 returned error can't find the container with id 7b1d15b5824ff965dbae0db96da7e1fb888a030605c99b050d2408114312b3b1
Sep 06 04:55:27.644358 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:27.644316    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da02b5e_ca23_4376_9c84_7b1ff2713408.slice/crio-1d3b9fcfe361f21ad83aec24fb79a669e52f990f2e2dbaae2c2ee884de737de2 WatchSource:0}: Error finding container 1d3b9fcfe361f21ad83aec24fb79a669e52f990f2e2dbaae2c2ee884de737de2: Status 404 returned error can't find the container with id 1d3b9fcfe361f21ad83aec24fb79a669e52f990f2e2dbaae2c2ee884de737de2
Sep 06 04:55:27.645918 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:27.645895    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2153f522_a60a_4c12_8aea_bf25f12e162e.slice/crio-d6f37092f9946880402eb42f7c37cf2e97c9b3e25413cfd41676907f6f975020 WatchSource:0}: Error finding container d6f37092f9946880402eb42f7c37cf2e97c9b3e25413cfd41676907f6f975020: Status 404 returned error can't find the container with id d6f37092f9946880402eb42f7c37cf2e97c9b3e25413cfd41676907f6f975020
Sep 06 04:55:27.646531 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:27.646507    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53aa3ac4_3e77_40df_9136_ff0e41c2606b.slice/crio-0d0e922f0fdcad2cd529b30dfc942e9159a699c767a94af21d270528ba7d482e WatchSource:0}: Error finding container 0d0e922f0fdcad2cd529b30dfc942e9159a699c767a94af21d270528ba7d482e: Status 404 returned error can't find the container with id 0d0e922f0fdcad2cd529b30dfc942e9159a699c767a94af21d270528ba7d482e
Sep 06 04:55:27.647412 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:27.647391    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf6fbec_f0bf_44ba_99dd_355169f9da5e.slice/crio-a17592b7d36522b41e0807106faabd0966351e3302c909c9334e17861902dab8 WatchSource:0}: Error finding container a17592b7d36522b41e0807106faabd0966351e3302c909c9334e17861902dab8: Status 404 returned error can't find the container with id a17592b7d36522b41e0807106faabd0966351e3302c909c9334e17861902dab8
Sep 06 04:55:27.648880 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:27.648817    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb6a418_6784_4b50_bfc4_f8331eb404f1.slice/crio-6591b165d9d60b2bbe845ac0eb9280b8d5b2e7d9f7466d80e658196b97376c3e WatchSource:0}: Error finding container 6591b165d9d60b2bbe845ac0eb9280b8d5b2e7d9f7466d80e658196b97376c3e: Status 404 returned error can't find the container with id 6591b165d9d60b2bbe845ac0eb9280b8d5b2e7d9f7466d80e658196b97376c3e
Sep 06 04:55:27.652652 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:55:27.652630    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfa2398_7840_4bc9_b5d1_61bc9e8cfc96.slice/crio-504f915db7b245d0dbfc92bfcc1bd38790f4574e66f00ece290d1b593e5ed8cd WatchSource:0}: Error finding container 504f915db7b245d0dbfc92bfcc1bd38790f4574e66f00ece290d1b593e5ed8cd: Status 404 returned error can't find the container with id 504f915db7b245d0dbfc92bfcc1bd38790f4574e66f00ece290d1b593e5ed8cd
Sep 06 04:55:27.905948 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:27.905913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7wg\" (UniqueName: \"kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg\") pod \"network-check-target-cqc98\" (UID: \"6ce9940b-e5b0-4647-bf32-532f1806d6a3\") " pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:27.906092 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:27.906064    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 04:55:27.906092 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:27.906082    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 04:55:27.906092 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:27.906092    2580 projected.go:196] Error preparing data for projected volume kube-api-access-zq7wg for pod openshift-network-diagnostics/network-check-target-cqc98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:27.906246 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:27.906148    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg podName:6ce9940b-e5b0-4647-bf32-532f1806d6a3 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:29.906134085 +0000 UTC m=+6.047694008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zq7wg" (UniqueName: "kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg") pod "network-check-target-cqc98" (UID: "6ce9940b-e5b0-4647-bf32-532f1806d6a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:28.006223 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.006198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:28.006333 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.006253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:28.006375 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:28.006359    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:28.006375 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:28.006363    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:28.006433 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:28.006415    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret podName:d08567e7-7463-408b-9d6f-d6edca57b781 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:30.006397699 +0000 UTC m=+6.147957621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret") pod "global-pull-secret-syncer-g6f2s" (UID: "d08567e7-7463-408b-9d6f-d6edca57b781") : object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:28.006433 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:28.006429    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs podName:d3a5a3f7-dc34-4f63-9fa9-5660515f8000 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:30.006422358 +0000 UTC m=+6.147982277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs") pod "network-metrics-daemon-csrcb" (UID: "d3a5a3f7-dc34-4f63-9fa9-5660515f8000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:28.343254 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.343143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:28.343414 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:28.343322    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:28.344198 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.343692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:28.344198 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:28.343786    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:28.344198 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.343848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:28.344198 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:28.343916    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:28.359370 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.359341    2580 generic.go:356] "Generic (PLEG): container finished" podID="536e7fcc5b50e39897826a8d67dc9141" containerID="1d61d184e0fd06cad6763fa39f2fd7532bb066a3532f45332f2512da9da08120" exitCode=0
Sep 06 04:55:28.359965 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.359430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal" event={"ID":"536e7fcc5b50e39897826a8d67dc9141","Type":"ContainerDied","Data":"1d61d184e0fd06cad6763fa39f2fd7532bb066a3532f45332f2512da9da08120"}
Sep 06 04:55:28.361549 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.361525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vrnfr" event={"ID":"ebf6fbec-f0bf-44ba-99dd-355169f9da5e","Type":"ContainerStarted","Data":"a17592b7d36522b41e0807106faabd0966351e3302c909c9334e17861902dab8"}
Sep 06 04:55:28.379845 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.379799    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 04:50:26 +0000 UTC" deadline="2028-03-18 18:37:43.189762566 +0000 UTC"
Sep 06 04:55:28.379935 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.379847    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13429h42m14.809919917s"
Sep 06 04:55:28.388355 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.388294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d24f8" event={"ID":"2153f522-a60a-4c12-8aea-bf25f12e162e","Type":"ContainerStarted","Data":"d6f37092f9946880402eb42f7c37cf2e97c9b3e25413cfd41676907f6f975020"}
Sep 06 04:55:28.391685 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.391657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hr7nb" event={"ID":"53aa3ac4-3e77-40df-9136-ff0e41c2606b","Type":"ContainerStarted","Data":"0d0e922f0fdcad2cd529b30dfc942e9159a699c767a94af21d270528ba7d482e"}
Sep 06 04:55:28.399546 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.399513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2" event={"ID":"7d8c10d8-49ae-4bb5-8083-6c2596d4321a","Type":"ContainerStarted","Data":"6c13569ab2480b0246ecf1a7feb3d3b6650251b1ef784b81c46e747864f1a75b"}
Sep 06 04:55:28.403916 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.403825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" event={"ID":"7da02b5e-ca23-4376-9c84-7b1ff2713408","Type":"ContainerStarted","Data":"1d3b9fcfe361f21ad83aec24fb79a669e52f990f2e2dbaae2c2ee884de737de2"}
Sep 06 04:55:28.414899 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.414673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-42d2r" event={"ID":"4bb6a418-6784-4b50-bfc4-f8331eb404f1","Type":"ContainerStarted","Data":"6591b165d9d60b2bbe845ac0eb9280b8d5b2e7d9f7466d80e658196b97376c3e"}
Sep 06 04:55:28.426027 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.426002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kwqp6" event={"ID":"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96","Type":"ContainerStarted","Data":"504f915db7b245d0dbfc92bfcc1bd38790f4574e66f00ece290d1b593e5ed8cd"}
Sep 06 04:55:28.432745 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.432719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vgwr4" event={"ID":"93f31079-87fb-497d-bdc4-5ae1b391f5ad","Type":"ContainerStarted","Data":"7b1d15b5824ff965dbae0db96da7e1fb888a030605c99b050d2408114312b3b1"}
Sep 06 04:55:28.438668 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.438644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal" event={"ID":"2a6422a356691d2cbb645b3d74be5d53","Type":"ContainerStarted","Data":"5b408ca8edf70728bc3f7188734ba20a25bafe098661d12732d466bf42e7183e"}
Sep 06 04:55:28.453480 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:28.452452    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal" podStartSLOduration=3.452436806 podStartE2EDuration="3.452436806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:55:28.452331448 +0000 UTC m=+4.593891391" watchObservedRunningTime="2026-09-06 04:55:28.452436806 +0000 UTC m=+4.593996750"
Sep 06 04:55:29.456867 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:29.456822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal" event={"ID":"536e7fcc5b50e39897826a8d67dc9141","Type":"ContainerStarted","Data":"6de0862e373421b68727bfd13667a6e5fe4a80dab92b878138de2c30b3287cce"}
Sep 06 04:55:29.921438 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:29.921319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7wg\" (UniqueName: \"kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg\") pod \"network-check-target-cqc98\" (UID: \"6ce9940b-e5b0-4647-bf32-532f1806d6a3\") " pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:29.921584 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:29.921542    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 04:55:29.921584 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:29.921560    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 04:55:29.921584 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:29.921574    2580 projected.go:196] Error preparing data for projected volume kube-api-access-zq7wg for pod openshift-network-diagnostics/network-check-target-cqc98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:29.921848 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:29.921733    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg podName:6ce9940b-e5b0-4647-bf32-532f1806d6a3 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:33.921712396 +0000 UTC m=+10.063272338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zq7wg" (UniqueName: "kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg") pod "network-check-target-cqc98" (UID: "6ce9940b-e5b0-4647-bf32-532f1806d6a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:30.022582 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:30.022535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:30.022763 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:30.022618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:30.022763 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:30.022748    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:30.022875 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:30.022804    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret podName:d08567e7-7463-408b-9d6f-d6edca57b781 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:34.022786293 +0000 UTC m=+10.164346214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret") pod "global-pull-secret-syncer-g6f2s" (UID: "d08567e7-7463-408b-9d6f-d6edca57b781") : object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:30.023213 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:30.023194    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:30.023322 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:30.023251    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs podName:d3a5a3f7-dc34-4f63-9fa9-5660515f8000 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:34.023235222 +0000 UTC m=+10.164795146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs") pod "network-metrics-daemon-csrcb" (UID: "d3a5a3f7-dc34-4f63-9fa9-5660515f8000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:30.342363 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:30.342258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:30.342518 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:30.342385    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:30.342800 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:30.342780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:30.342914 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:30.342891    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:30.343684 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:30.343501    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:30.343684 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:30.343595    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:32.342277 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:32.341199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:32.342277 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:32.341338    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:32.342277 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:32.341719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:32.342277 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:32.341807    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:32.342277 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:32.342135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:32.342277 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:32.342240    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:33.954489 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:33.954459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7wg\" (UniqueName: \"kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg\") pod \"network-check-target-cqc98\" (UID: \"6ce9940b-e5b0-4647-bf32-532f1806d6a3\") " pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:33.954990 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:33.954596    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 04:55:33.954990 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:33.954615    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 04:55:33.954990 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:33.954624    2580 projected.go:196] Error preparing data for projected volume kube-api-access-zq7wg for pod openshift-network-diagnostics/network-check-target-cqc98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:33.954990 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:33.954678    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg podName:6ce9940b-e5b0-4647-bf32-532f1806d6a3 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:41.95466111 +0000 UTC m=+18.096221047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zq7wg" (UniqueName: "kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg") pod "network-check-target-cqc98" (UID: "6ce9940b-e5b0-4647-bf32-532f1806d6a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:34.055635 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:34.055582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:34.055794 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:34.055673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:34.055794 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:34.055781    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:34.055907 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:34.055843    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret podName:d08567e7-7463-408b-9d6f-d6edca57b781 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:42.055825701 +0000 UTC m=+18.197385642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret") pod "global-pull-secret-syncer-g6f2s" (UID: "d08567e7-7463-408b-9d6f-d6edca57b781") : object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:34.055970 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:34.055914    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:34.055970 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:34.055949    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs podName:d3a5a3f7-dc34-4f63-9fa9-5660515f8000 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:42.055936641 +0000 UTC m=+18.197496566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs") pod "network-metrics-daemon-csrcb" (UID: "d3a5a3f7-dc34-4f63-9fa9-5660515f8000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:34.342145 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:34.342030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:34.342318 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:34.342186    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:34.342318 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:34.342204    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:34.342424 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:34.342313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:34.342424 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:34.342369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:34.342539 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:34.342432    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:36.342221 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:36.341782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:36.342221 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:36.341781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:36.342221 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:36.341907    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:36.342221 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:36.342003    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:36.342221 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:36.342049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:36.342221 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:36.342120    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:38.341051 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:38.340971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:38.341051 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:38.340991    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:38.341486 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:38.340972    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:38.341486 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:38.341088    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:38.341486 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:38.341201    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:38.341486 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:38.341304    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:40.344110 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:40.344077    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:40.344557 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:40.344205    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:40.344557 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:40.344216    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:40.344557 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:40.344233    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:40.344557 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:40.344341    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:40.344557 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:40.344452    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:42.013835 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:42.013798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7wg\" (UniqueName: \"kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg\") pod \"network-check-target-cqc98\" (UID: \"6ce9940b-e5b0-4647-bf32-532f1806d6a3\") " pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:42.014367 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.013992    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 04:55:42.014367 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.014016    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 04:55:42.014367 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.014026    2580 projected.go:196] Error preparing data for projected volume kube-api-access-zq7wg for pod openshift-network-diagnostics/network-check-target-cqc98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:42.014367 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.014077    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg podName:6ce9940b-e5b0-4647-bf32-532f1806d6a3 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:58.014063029 +0000 UTC m=+34.155622954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zq7wg" (UniqueName: "kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg") pod "network-check-target-cqc98" (UID: "6ce9940b-e5b0-4647-bf32-532f1806d6a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:42.114365 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:42.114328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:42.114539 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:42.114407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:42.114539 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.114447    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:42.114539 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.114487    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:42.114539 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.114523    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs podName:d3a5a3f7-dc34-4f63-9fa9-5660515f8000 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:58.114502584 +0000 UTC m=+34.256062509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs") pod "network-metrics-daemon-csrcb" (UID: "d3a5a3f7-dc34-4f63-9fa9-5660515f8000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:42.114698 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.114542    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret podName:d08567e7-7463-408b-9d6f-d6edca57b781 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:58.114532233 +0000 UTC m=+34.256092160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret") pod "global-pull-secret-syncer-g6f2s" (UID: "d08567e7-7463-408b-9d6f-d6edca57b781") : object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:42.345012 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:42.344941    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:42.345012 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:42.344955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:42.345012 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:42.344955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:42.345304 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.345057    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:42.345304 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.345119    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:42.345304 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:42.345243    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:44.341959 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:44.341927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:44.341959 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:44.341960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:44.342475 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:44.342073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:44.342606 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:44.342574    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:44.342677 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:44.342644    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:44.342730 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:44.342690    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:46.341280 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.341016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:46.341863 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.341017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:46.341863 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.341055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:46.341863 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:46.341399    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:46.341863 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:46.341534    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:46.341863 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:46.341617    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:46.493990 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.493958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d24f8" event={"ID":"2153f522-a60a-4c12-8aea-bf25f12e162e","Type":"ContainerStarted","Data":"57707f530bca958827c3f02a188b236ab17e8e3cf21ccc9316a6e51d0774d06a"}
Sep 06 04:55:46.495363 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.495340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hr7nb" event={"ID":"53aa3ac4-3e77-40df-9136-ff0e41c2606b","Type":"ContainerStarted","Data":"cafd66666424f9282ea2dd270fac8042aaabe6a73aaf6c23fa3c798870da5c11"}
Sep 06 04:55:46.496819 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.496789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2" event={"ID":"7d8c10d8-49ae-4bb5-8083-6c2596d4321a","Type":"ContainerStarted","Data":"7a34f8b453c5b01ea5f1dbd44a900e6d59693c1760f0feda8ea1c8e17f809575"}
Sep 06 04:55:46.499494 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.499477    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 04:55:46.499827 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.499809    2580 generic.go:356] "Generic (PLEG): container finished" podID="7da02b5e-ca23-4376-9c84-7b1ff2713408" containerID="02d5914f357e1e1c07399b55376f2091f4091b1b30a5753e9e7883f8ad8f5752" exitCode=1
Sep 06 04:55:46.499892 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.499877    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" event={"ID":"7da02b5e-ca23-4376-9c84-7b1ff2713408","Type":"ContainerStarted","Data":"9d9dda601aa13cec5c007ee70f9eff8ab67abac146798fb582f6ec7a29444632"}
Sep 06 04:55:46.499954 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.499940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" event={"ID":"7da02b5e-ca23-4376-9c84-7b1ff2713408","Type":"ContainerStarted","Data":"29740823fc5ccc209e8d6783647f4cbaa280e434620b1f9d822ad7658a7ce3a7"}
Sep 06 04:55:46.499995 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.499985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" event={"ID":"7da02b5e-ca23-4376-9c84-7b1ff2713408","Type":"ContainerStarted","Data":"b3a79fa06185efdff29e5072b7baab3bf328527271a843716e92d8521e78b86e"}
Sep 06 04:55:46.500029 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.500016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" event={"ID":"7da02b5e-ca23-4376-9c84-7b1ff2713408","Type":"ContainerStarted","Data":"5937b17d7878986147f7bc6df5da0fda0ce406d91e02fa87853305478537e69a"}
Sep 06 04:55:46.500071 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.500047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" event={"ID":"7da02b5e-ca23-4376-9c84-7b1ff2713408","Type":"ContainerDied","Data":"02d5914f357e1e1c07399b55376f2091f4091b1b30a5753e9e7883f8ad8f5752"}
Sep 06 04:55:46.500115 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.500080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" event={"ID":"7da02b5e-ca23-4376-9c84-7b1ff2713408","Type":"ContainerStarted","Data":"47ba5fff23ccf7b7cf70a072769447e1dd73b4ca1917b3c807d69cf8150a5e87"}
Sep 06 04:55:46.501270 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.501250    2580 generic.go:356] "Generic (PLEG): container finished" podID="a99b800d-76a9-4349-82f8-55804daefcde" containerID="72d742ee9dbb9b73647d137f89ab61529e1aad69c8632bc27998f56313bbc4a0" exitCode=0
Sep 06 04:55:46.501350 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.501321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4568p" event={"ID":"a99b800d-76a9-4349-82f8-55804daefcde","Type":"ContainerDied","Data":"72d742ee9dbb9b73647d137f89ab61529e1aad69c8632bc27998f56313bbc4a0"}
Sep 06 04:55:46.502805 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.502779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kwqp6" event={"ID":"fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96","Type":"ContainerStarted","Data":"e213c58659c829b94f9b419326b1392373d712e84ec65abfb800bd269152c722"}
Sep 06 04:55:46.504320 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.504300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vgwr4" event={"ID":"93f31079-87fb-497d-bdc4-5ae1b391f5ad","Type":"ContainerStarted","Data":"c189f9af3c52ba35f116847861e040edc9751fddef63a75eebc9d430a4f22d91"}
Sep 06 04:55:46.505679 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.505649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vrnfr" event={"ID":"ebf6fbec-f0bf-44ba-99dd-355169f9da5e","Type":"ContainerStarted","Data":"b3e604125f05e1e577c27c520a576e9b42ded733bf9958a538dad87af0ee7770"}
Sep 06 04:55:46.506728 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.506694    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d24f8" podStartSLOduration=4.390980503 podStartE2EDuration="22.506683465s" totalImagesPullingTime="18.115702962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:55:46.506596224 +0000 UTC m=+22.648156166" watchObservedRunningTime="2026-09-06 04:55:46.506683465 +0000 UTC m=+22.648243407"
Sep 06 04:55:46.507107 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.507085    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal" podStartSLOduration=21.507079302 podStartE2EDuration="21.507079302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:55:29.47108918 +0000 UTC m=+5.612649125" watchObservedRunningTime="2026-09-06 04:55:46.507079302 +0000 UTC m=+22.648639244"
Sep 06 04:55:46.535552 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.535519    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hr7nb" podStartSLOduration=4.57027832 podStartE2EDuration="22.535507979s" totalImagesPullingTime="17.965229659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:55:46.535185319 +0000 UTC m=+22.676745278" watchObservedRunningTime="2026-09-06 04:55:46.535507979 +0000 UTC m=+22.677067921"
Sep 06 04:55:46.548756 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.548725    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vrnfr" podStartSLOduration=4.022804087 podStartE2EDuration="22.548715528s" totalImagesPullingTime="18.525911441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:55:46.548544536 +0000 UTC m=+22.690104478" watchObservedRunningTime="2026-09-06 04:55:46.548715528 +0000 UTC m=+22.690275470"
Sep 06 04:55:46.562421 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.562389    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kwqp6" podStartSLOduration=4.451330163 podStartE2EDuration="22.562379323s" totalImagesPullingTime="18.11104916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:55:46.562180269 +0000 UTC m=+22.703740223" watchObservedRunningTime="2026-09-06 04:55:46.562379323 +0000 UTC m=+22.703939265"
Sep 06 04:55:46.577997 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:46.577966    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vgwr4" podStartSLOduration=4.458893686 podStartE2EDuration="22.57795596s" totalImagesPullingTime="18.119062274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:55:46.577827251 +0000 UTC m=+22.719387214" watchObservedRunningTime="2026-09-06 04:55:46.57795596 +0000 UTC m=+22.719515902"
Sep 06 04:55:47.510488 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:47.510438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-42d2r" event={"ID":"4bb6a418-6784-4b50-bfc4-f8331eb404f1","Type":"ContainerStarted","Data":"2e5e6bfbd8cdf8847be6563a80a60cff625fdcdfca85935ea150e425aa38f026"}
Sep 06 04:55:47.523944 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:47.523902    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-42d2r" podStartSLOduration=5.412652559 podStartE2EDuration="23.52388831s" totalImagesPullingTime="18.111235751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:55:47.523384234 +0000 UTC m=+23.664944195" watchObservedRunningTime="2026-09-06 04:55:47.52388831 +0000 UTC m=+23.665448252"
Sep 06 04:55:47.960892 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:47.960695    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 06 04:55:48.035608 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.035582    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:48.036220 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.036203    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:48.283988 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.283956    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:48.284578 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.284558    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d24f8"
Sep 06 04:55:48.337039 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.336929    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T04:55:47.960889246Z","UUID":"a4dc6e26-7f09-4841-8815-c8a7bbff10d3","Handler":null,"Name":"","Endpoint":""}
Sep 06 04:55:48.338342 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.338323    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 06 04:55:48.338444 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.338351    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 06 04:55:48.341580 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.341559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:48.341699 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.341554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:48.341699 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:48.341667    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:48.341699 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.341687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:48.341860 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:48.341778    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:48.341860 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:48.341848    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:48.514668 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.514639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2" event={"ID":"7d8c10d8-49ae-4bb5-8083-6c2596d4321a","Type":"ContainerStarted","Data":"7576f704e49615aa6a59e08ddf048fc9a7429ad30ca911c229138d3b7e99e4a4"}
Sep 06 04:55:48.518361 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.518342    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 04:55:48.518719 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:48.518691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" event={"ID":"7da02b5e-ca23-4376-9c84-7b1ff2713408","Type":"ContainerStarted","Data":"7bf280e2923f431a0c9e8d1203b43ff16a1bf20eb9256a0e2ed1f6642658f43e"}
Sep 06 04:55:49.523439 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:49.523401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2" event={"ID":"7d8c10d8-49ae-4bb5-8083-6c2596d4321a","Type":"ContainerStarted","Data":"08e78c6d0f41334ac2904ad1e1e11518c9183f8e93a23a47c6afeabde17bc9ab"}
Sep 06 04:55:49.539932 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:49.539884    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79zq2" podStartSLOduration=4.380340407 podStartE2EDuration="25.539869358s" totalImagesPullingTime="21.159528951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:55:49.539867147 +0000 UTC m=+25.681427109" watchObservedRunningTime="2026-09-06 04:55:49.539869358 +0000 UTC m=+25.681429300"
Sep 06 04:55:50.341230 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:50.341202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:50.341230 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:50.341218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:50.341453 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:50.341218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:50.341453 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:50.341292    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:50.341453 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:50.341383    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:50.341562 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:50.341453    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:52.345019 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.344869    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:52.345570 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.344923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:52.345570 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:52.345119    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:52.345570 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.344922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:52.345570 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:52.345215    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:52.345570 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:52.345244    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:52.536418 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.536398    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 04:55:52.536742 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.536720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" event={"ID":"7da02b5e-ca23-4376-9c84-7b1ff2713408","Type":"ContainerStarted","Data":"b7a7ac35bd8b44d9ac819caa7bf563fb6d8d7ebed9f8847d00c8ce01145ad4f3"}
Sep 06 04:55:52.536994 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.536970    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:52.537107 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.537048    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:52.537200 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.537188    2580 scope.go:117] "RemoveContainer" containerID="02d5914f357e1e1c07399b55376f2091f4091b1b30a5753e9e7883f8ad8f5752"
Sep 06 04:55:52.542229 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.542202    2580 generic.go:356] "Generic (PLEG): container finished" podID="a99b800d-76a9-4349-82f8-55804daefcde" containerID="bb3e335eb728a27261bc58fb03e2002797dfe77d875f9c0e145bda88d9088bfc" exitCode=0
Sep 06 04:55:52.542328 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.542246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4568p" event={"ID":"a99b800d-76a9-4349-82f8-55804daefcde","Type":"ContainerDied","Data":"bb3e335eb728a27261bc58fb03e2002797dfe77d875f9c0e145bda88d9088bfc"}
Sep 06 04:55:52.554091 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:52.554060    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:53.548202 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.548174    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 04:55:53.548844 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.548572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" event={"ID":"7da02b5e-ca23-4376-9c84-7b1ff2713408","Type":"ContainerStarted","Data":"c300d3f09c7c9bfd7073c2776f42dac000e6980c8cb57f0693d0748628313308"}
Sep 06 04:55:53.549087 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.549061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:53.564168 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.564131    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:55:53.575463 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.575422    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk" podStartSLOduration=11.45024712 podStartE2EDuration="29.575406198s" totalImagesPullingTime="18.125159078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:55:53.573761663 +0000 UTC m=+29.715321615" watchObservedRunningTime="2026-09-06 04:55:53.575406198 +0000 UTC m=+29.716966133"
Sep 06 04:55:53.992110 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.992080    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g6f2s"]
Sep 06 04:55:53.992273 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.992259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:53.992392 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:53.992370    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:53.992950 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.992924    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cqc98"]
Sep 06 04:55:53.993062 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.993037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:53.993183 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:53.993146    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:53.993539 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.993517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-csrcb"]
Sep 06 04:55:53.993622 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:53.993610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:53.993734 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:53.993713    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:54.553491 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:54.553309    2580 generic.go:356] "Generic (PLEG): container finished" podID="a99b800d-76a9-4349-82f8-55804daefcde" containerID="3c18776b6ff30afce66c2d0b20863aabb84c4f9746afc50b0b9150327dbcf36f" exitCode=0
Sep 06 04:55:54.553932 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:54.553393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4568p" event={"ID":"a99b800d-76a9-4349-82f8-55804daefcde","Type":"ContainerDied","Data":"3c18776b6ff30afce66c2d0b20863aabb84c4f9746afc50b0b9150327dbcf36f"}
Sep 06 04:55:55.340927 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:55.340903    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:55.341037 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:55.341015    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:55.557713 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:55.557646    2580 generic.go:356] "Generic (PLEG): container finished" podID="a99b800d-76a9-4349-82f8-55804daefcde" containerID="79ce039a2cc96ed77207d244794998d24f7953e1688068896ae5c1ba557092a2" exitCode=0
Sep 06 04:55:55.558011 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:55.557723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4568p" event={"ID":"a99b800d-76a9-4349-82f8-55804daefcde","Type":"ContainerDied","Data":"79ce039a2cc96ed77207d244794998d24f7953e1688068896ae5c1ba557092a2"}
Sep 06 04:55:56.341122 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:56.341093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:56.341298 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:56.341247    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:56.341368 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:56.341304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:56.341449 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:56.341404    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:57.340855 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:57.340820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:57.341381 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:57.340941    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc98" podUID="6ce9940b-e5b0-4647-bf32-532f1806d6a3"
Sep 06 04:55:58.028167 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.028114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7wg\" (UniqueName: \"kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg\") pod \"network-check-target-cqc98\" (UID: \"6ce9940b-e5b0-4647-bf32-532f1806d6a3\") " pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:58.028327 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.028307    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 04:55:58.028371 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.028330    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 04:55:58.028371 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.028341    2580 projected.go:196] Error preparing data for projected volume kube-api-access-zq7wg for pod openshift-network-diagnostics/network-check-target-cqc98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:58.028432 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.028393    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg podName:6ce9940b-e5b0-4647-bf32-532f1806d6a3 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:30.028379522 +0000 UTC m=+66.169939442 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zq7wg" (UniqueName: "kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg") pod "network-check-target-cqc98" (UID: "6ce9940b-e5b0-4647-bf32-532f1806d6a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 04:55:58.128704 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.128667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:58.128879 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.128743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:58.128879 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.128830    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:58.128879 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.128853    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:58.129041 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.128891    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs podName:d3a5a3f7-dc34-4f63-9fa9-5660515f8000 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:30.128877583 +0000 UTC m=+66.270437503 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs") pod "network-metrics-daemon-csrcb" (UID: "d3a5a3f7-dc34-4f63-9fa9-5660515f8000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 04:55:58.129041 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.128906    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret podName:d08567e7-7463-408b-9d6f-d6edca57b781 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:30.128900228 +0000 UTC m=+66.270460148 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret") pod "global-pull-secret-syncer-g6f2s" (UID: "d08567e7-7463-408b-9d6f-d6edca57b781") : object "kube-system"/"original-pull-secret" not registered
Sep 06 04:55:58.341857 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.341781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:55:58.342294 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.341905    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6f2s" podUID="d08567e7-7463-408b-9d6f-d6edca57b781"
Sep 06 04:55:58.342294 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.341949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:55:58.342294 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.342062    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:55:58.686327 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.686299    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeReady"
Sep 06 04:55:58.686451 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.686423    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 06 04:55:58.719393 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.719361    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-dtkkv"]
Sep 06 04:55:58.745203 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.745171    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f689b587d-grvvf"]
Sep 06 04:55:58.745359 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.745253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:55:58.747491 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.747463    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 06 04:55:58.747607 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.747539    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 06 04:55:58.747607 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.747543    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wzm9q\""
Sep 06 04:55:58.760900 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.760879    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mx4tj"]
Sep 06 04:55:58.760997 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.760961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:58.763060 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.763041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 06 04:55:58.763150 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.763076    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fgl4q\""
Sep 06 04:55:58.763346 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.763333    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 06 04:55:58.763393 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.763375    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 06 04:55:58.768149 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.768126    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 06 04:55:58.781216 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.781193    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-dtkkv"]
Sep 06 04:55:58.781295 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.781221    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-55l5h"]
Sep 06 04:55:58.781295 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.781289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:58.783448 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.783425    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 06 04:55:58.783536 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.783461    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-prh2m\""
Sep 06 04:55:58.783536 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.783499    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 06 04:55:58.802187 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.802147    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f689b587d-grvvf"]
Sep 06 04:55:58.802283 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.802199    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mx4tj"]
Sep 06 04:55:58.802283 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.802218    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-55l5h"]
Sep 06 04:55:58.802283 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.802236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-55l5h"
Sep 06 04:55:58.804542 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.804522    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 06 04:55:58.804641 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.804544    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7h5qb\""
Sep 06 04:55:58.804641 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.804559    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 06 04:55:58.804755 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.804641    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 06 04:55:58.833938 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.833916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c64b1fde-c4e4-49b4-a005-7e721932b165-nginx-conf\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:55:58.834065 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.834032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:55:58.935360 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:58.935360 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzh4\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-kube-api-access-zjzh4\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:58.935360 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-tmp-dir\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:58.935576 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:55:58.935576 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c64b1fde-c4e4-49b4-a005-7e721932b165-nginx-conf\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:55:58.935576 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-ca-trust-extracted\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:58.935576 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3-cert\") pod \"ingress-canary-55l5h\" (UID: \"8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3\") " pod="openshift-ingress-canary/ingress-canary-55l5h"
Sep 06 04:55:58.935757 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-bound-sa-token\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:58.935757 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.935610    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 06 04:55:58.935757 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x462f\" (UniqueName: \"kubernetes.io/projected/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-kube-api-access-x462f\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:58.935757 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-certificates\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:58.935757 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:58.935683    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert podName:c64b1fde-c4e4-49b4-a005-7e721932b165 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:59.435665345 +0000 UTC m=+35.577225267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-dtkkv" (UID: "c64b1fde-c4e4-49b4-a005-7e721932b165") : secret "networking-console-plugin-cert" not found
Sep 06 04:55:58.935757 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-installation-pull-secrets\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:58.935757 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7r6\" (UniqueName: \"kubernetes.io/projected/8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3-kube-api-access-pb7r6\") pod \"ingress-canary-55l5h\" (UID: \"8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3\") " pod="openshift-ingress-canary/ingress-canary-55l5h"
Sep 06 04:55:58.936014 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-image-registry-private-configuration\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:58.936014 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-config-volume\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:58.936014 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:58.936014 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.935912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-trusted-ca\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:58.936391 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:58.936370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c64b1fde-c4e4-49b4-a005-7e721932b165-nginx-conf\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:55:59.036451 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-ca-trust-extracted\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.036451 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3-cert\") pod \"ingress-canary-55l5h\" (UID: \"8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3\") " pod="openshift-ingress-canary/ingress-canary-55l5h"
Sep 06 04:55:59.036688 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-bound-sa-token\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.036688 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x462f\" (UniqueName: \"kubernetes.io/projected/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-kube-api-access-x462f\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:59.036688 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-certificates\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.036688 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-installation-pull-secrets\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.036688 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7r6\" (UniqueName: \"kubernetes.io/projected/8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3-kube-api-access-pb7r6\") pod \"ingress-canary-55l5h\" (UID: \"8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3\") " pod="openshift-ingress-canary/ingress-canary-55l5h"
Sep 06 04:55:59.036688 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-image-registry-private-configuration\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.036688 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-config-volume\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:59.036688 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:59.037064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-trusted-ca\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.037064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.037064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzh4\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-kube-api-access-zjzh4\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.037064 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.036836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-ca-trust-extracted\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.041311 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.041287    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 04:55:59.041311 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.041312    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f689b587d-grvvf: secret "image-registry-tls" not found
Sep 06 04:55:59.041514 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.041376    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls podName:531caa8b-f2b0-41f1-9c91-b68f8fbbfa33 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:59.541357331 +0000 UTC m=+35.682917261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls") pod "image-registry-6f689b587d-grvvf" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33") : secret "image-registry-tls" not found
Sep 06 04:55:59.043662 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.041702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-certificates\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.043662 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.042139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-config-volume\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:59.043662 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.042398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-image-registry-private-configuration\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.043662 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.042786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3-cert\") pod \"ingress-canary-55l5h\" (UID: \"8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3\") " pod="openshift-ingress-canary/ingress-canary-55l5h"
Sep 06 04:55:59.043662 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.043041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-trusted-ca\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.043662 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.043491    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 04:55:59.043662 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.043551    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls podName:84a76bc6-6f0f-4d5c-b48f-9e8ab6651556 nodeName:}" failed. No retries permitted until 2026-09-06 04:55:59.543533543 +0000 UTC m=+35.685093463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls") pod "dns-default-mx4tj" (UID: "84a76bc6-6f0f-4d5c-b48f-9e8ab6651556") : secret "dns-default-metrics-tls" not found
Sep 06 04:55:59.044261 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.044202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-tmp-dir\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:59.044581 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.044558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-tmp-dir\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:59.046349 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.045871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzh4\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-kube-api-access-zjzh4\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.047196 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.047173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-bound-sa-token\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.047314 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.047256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-installation-pull-secrets\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.050386 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.050340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x462f\" (UniqueName: \"kubernetes.io/projected/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-kube-api-access-x462f\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:59.050386 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.050363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7r6\" (UniqueName: \"kubernetes.io/projected/8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3-kube-api-access-pb7r6\") pod \"ingress-canary-55l5h\" (UID: \"8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3\") " pod="openshift-ingress-canary/ingress-canary-55l5h"
Sep 06 04:55:59.111406 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.111189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-55l5h"
Sep 06 04:55:59.341129 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.341106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:55:59.344045 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.344014    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 04:55:59.344045 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.344016    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dzjkx\""
Sep 06 04:55:59.344449 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.344016    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 04:55:59.448465 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.448421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:55:59.448623 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.448570    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 06 04:55:59.448690 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.448646    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert podName:c64b1fde-c4e4-49b4-a005-7e721932b165 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:00.448625383 +0000 UTC m=+36.590185303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-dtkkv" (UID: "c64b1fde-c4e4-49b4-a005-7e721932b165") : secret "networking-console-plugin-cert" not found
Sep 06 04:55:59.549634 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.549602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:55:59.549802 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:55:59.549661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:55:59.549802 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.549765    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 04:55:59.549802 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.549780    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 04:55:59.549802 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.549795    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f689b587d-grvvf: secret "image-registry-tls" not found
Sep 06 04:55:59.550003 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.549844    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls podName:84a76bc6-6f0f-4d5c-b48f-9e8ab6651556 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:00.549823342 +0000 UTC m=+36.691383288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls") pod "dns-default-mx4tj" (UID: "84a76bc6-6f0f-4d5c-b48f-9e8ab6651556") : secret "dns-default-metrics-tls" not found
Sep 06 04:55:59.550003 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:55:59.549863    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls podName:531caa8b-f2b0-41f1-9c91-b68f8fbbfa33 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:00.549854166 +0000 UTC m=+36.691414091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls") pod "image-registry-6f689b587d-grvvf" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33") : secret "image-registry-tls" not found
Sep 06 04:56:00.341109 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:00.341070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:56:00.341288 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:00.341120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:56:00.343585 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:00.343563    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b9kvk\""
Sep 06 04:56:00.343714 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:00.343559    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 04:56:00.344376 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:00.344360    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 04:56:00.455756 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:00.455723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:56:00.455908 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:00.455883    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 06 04:56:00.455987 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:00.455977    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert podName:c64b1fde-c4e4-49b4-a005-7e721932b165 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:02.455953797 +0000 UTC m=+38.597513744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-dtkkv" (UID: "c64b1fde-c4e4-49b4-a005-7e721932b165") : secret "networking-console-plugin-cert" not found
Sep 06 04:56:00.556172 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:00.556123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:56:00.556343 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:00.556192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:56:00.556343 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:00.556284    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 04:56:00.556343 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:00.556309    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 04:56:00.556343 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:00.556319    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f689b587d-grvvf: secret "image-registry-tls" not found
Sep 06 04:56:00.556483 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:00.556351    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls podName:84a76bc6-6f0f-4d5c-b48f-9e8ab6651556 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:02.556334255 +0000 UTC m=+38.697894176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls") pod "dns-default-mx4tj" (UID: "84a76bc6-6f0f-4d5c-b48f-9e8ab6651556") : secret "dns-default-metrics-tls" not found
Sep 06 04:56:00.556483 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:00.556365    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls podName:531caa8b-f2b0-41f1-9c91-b68f8fbbfa33 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:02.55635945 +0000 UTC m=+38.697919370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls") pod "image-registry-6f689b587d-grvvf" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33") : secret "image-registry-tls" not found
Sep 06 04:56:00.665881 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:00.665854    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-55l5h"]
Sep 06 04:56:00.669997 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:56:00.669968    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b09e0a1_fcf5_475c_bbfa_66fcf9c7eff3.slice/crio-1b3a6a7855efd9ffaaedf5766314a7de3c8725f14ec041d00c3600905734954c WatchSource:0}: Error finding container 1b3a6a7855efd9ffaaedf5766314a7de3c8725f14ec041d00c3600905734954c: Status 404 returned error can't find the container with id 1b3a6a7855efd9ffaaedf5766314a7de3c8725f14ec041d00c3600905734954c
Sep 06 04:56:01.574083 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:01.574039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-55l5h" event={"ID":"8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3","Type":"ContainerStarted","Data":"1b3a6a7855efd9ffaaedf5766314a7de3c8725f14ec041d00c3600905734954c"}
Sep 06 04:56:01.577257 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:01.577229    2580 generic.go:356] "Generic (PLEG): container finished" podID="a99b800d-76a9-4349-82f8-55804daefcde" containerID="09b1ab793f8814fad6d6e7b818924072905f11026dc803bcce0cc8a3ebde68ae" exitCode=0
Sep 06 04:56:01.577383 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:01.577281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4568p" event={"ID":"a99b800d-76a9-4349-82f8-55804daefcde","Type":"ContainerDied","Data":"09b1ab793f8814fad6d6e7b818924072905f11026dc803bcce0cc8a3ebde68ae"}
Sep 06 04:56:02.469822 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:02.469788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:56:02.469993 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:02.469960    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 06 04:56:02.470066 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:02.470042    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert podName:c64b1fde-c4e4-49b4-a005-7e721932b165 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:06.470019029 +0000 UTC m=+42.611578951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-dtkkv" (UID: "c64b1fde-c4e4-49b4-a005-7e721932b165") : secret "networking-console-plugin-cert" not found
Sep 06 04:56:02.570433 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:02.570400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:56:02.570602 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:02.570531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:56:02.570668 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:02.570599    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 04:56:02.570668 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:02.570619    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f689b587d-grvvf: secret "image-registry-tls" not found
Sep 06 04:56:02.570762 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:02.570667    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 04:56:02.570762 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:02.570681    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls podName:531caa8b-f2b0-41f1-9c91-b68f8fbbfa33 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:06.570658146 +0000 UTC m=+42.712218065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls") pod "image-registry-6f689b587d-grvvf" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33") : secret "image-registry-tls" not found
Sep 06 04:56:02.570762 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:02.570714    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls podName:84a76bc6-6f0f-4d5c-b48f-9e8ab6651556 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:06.570701753 +0000 UTC m=+42.712261672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls") pod "dns-default-mx4tj" (UID: "84a76bc6-6f0f-4d5c-b48f-9e8ab6651556") : secret "dns-default-metrics-tls" not found
Sep 06 04:56:02.584023 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:02.583993    2580 generic.go:356] "Generic (PLEG): container finished" podID="a99b800d-76a9-4349-82f8-55804daefcde" containerID="ecb490789597fea27f59cc5183a38069391bfb92ad76a1f159328ba0901bd050" exitCode=0
Sep 06 04:56:02.584476 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:02.584036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4568p" event={"ID":"a99b800d-76a9-4349-82f8-55804daefcde","Type":"ContainerDied","Data":"ecb490789597fea27f59cc5183a38069391bfb92ad76a1f159328ba0901bd050"}
Sep 06 04:56:03.587801 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:03.587766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-55l5h" event={"ID":"8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3","Type":"ContainerStarted","Data":"97b4244cf0c1a9d4e752facc54a9f5ebdc04e23f6d8886a3906ebff6f75ae610"}
Sep 06 04:56:03.591109 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:03.591086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4568p" event={"ID":"a99b800d-76a9-4349-82f8-55804daefcde","Type":"ContainerStarted","Data":"26c86e50e6ce21dc6a70cf4569e9abaf221237a4a349f4f15611d958205d7248"}
Sep 06 04:56:03.604066 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:03.604028    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-55l5h" podStartSLOduration=3.02002448 podStartE2EDuration="5.604015143s" totalImagesPullingTime="2.583990663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:56:03.603695007 +0000 UTC m=+39.745254960" watchObservedRunningTime="2026-09-06 04:56:03.604015143 +0000 UTC m=+39.745575082"
Sep 06 04:56:03.620812 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:03.620749    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4568p" podStartSLOduration=7.77559331 podStartE2EDuration="39.62073831s" totalImagesPullingTime="31.845145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:56:03.620077859 +0000 UTC m=+39.761637832" watchObservedRunningTime="2026-09-06 04:56:03.62073831 +0000 UTC m=+39.762298245"
Sep 06 04:56:06.498171 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:06.498129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:56:06.498567 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:06.498276    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 06 04:56:06.498567 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:06.498368    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert podName:c64b1fde-c4e4-49b4-a005-7e721932b165 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:14.498352396 +0000 UTC m=+50.639912319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-dtkkv" (UID: "c64b1fde-c4e4-49b4-a005-7e721932b165") : secret "networking-console-plugin-cert" not found
Sep 06 04:56:06.598982 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:06.598953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:56:06.599093 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:06.599029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:56:06.599131 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:06.599098    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 04:56:06.599131 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:06.599106    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 04:56:06.599131 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:06.599112    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f689b587d-grvvf: secret "image-registry-tls" not found
Sep 06 04:56:06.599250 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:06.599183    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls podName:84a76bc6-6f0f-4d5c-b48f-9e8ab6651556 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:14.599149299 +0000 UTC m=+50.740709218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls") pod "dns-default-mx4tj" (UID: "84a76bc6-6f0f-4d5c-b48f-9e8ab6651556") : secret "dns-default-metrics-tls" not found
Sep 06 04:56:06.599250 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:06.599200    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls podName:531caa8b-f2b0-41f1-9c91-b68f8fbbfa33 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:14.599191939 +0000 UTC m=+50.740751859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls") pod "image-registry-6f689b587d-grvvf" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33") : secret "image-registry-tls" not found
Sep 06 04:56:14.549598 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:14.549565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:56:14.550092 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:14.549677    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 06 04:56:14.550092 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:14.549722    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert podName:c64b1fde-c4e4-49b4-a005-7e721932b165 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:30.549710176 +0000 UTC m=+66.691270095 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-dtkkv" (UID: "c64b1fde-c4e4-49b4-a005-7e721932b165") : secret "networking-console-plugin-cert" not found
Sep 06 04:56:14.650692 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:14.650656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:56:14.650817 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:14.650703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:56:14.650817 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:14.650783    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 04:56:14.650887 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:14.650824    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 04:56:14.650887 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:14.650831    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls podName:84a76bc6-6f0f-4d5c-b48f-9e8ab6651556 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:30.65081871 +0000 UTC m=+66.792378629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls") pod "dns-default-mx4tj" (UID: "84a76bc6-6f0f-4d5c-b48f-9e8ab6651556") : secret "dns-default-metrics-tls" not found
Sep 06 04:56:14.650887 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:14.650836    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f689b587d-grvvf: secret "image-registry-tls" not found
Sep 06 04:56:14.650887 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:14.650869    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls podName:531caa8b-f2b0-41f1-9c91-b68f8fbbfa33 nodeName:}" failed. No retries permitted until 2026-09-06 04:56:30.650859511 +0000 UTC m=+66.792419431 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls") pod "image-registry-6f689b587d-grvvf" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33") : secret "image-registry-tls" not found
Sep 06 04:56:25.568372 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:25.568346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6gsk"
Sep 06 04:56:30.056521 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.056465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7wg\" (UniqueName: \"kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg\") pod \"network-check-target-cqc98\" (UID: \"6ce9940b-e5b0-4647-bf32-532f1806d6a3\") " pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:56:30.059332 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.059311    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 04:56:30.069229 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.069204    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 04:56:30.081336 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.081317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq7wg\" (UniqueName: \"kubernetes.io/projected/6ce9940b-e5b0-4647-bf32-532f1806d6a3-kube-api-access-zq7wg\") pod \"network-check-target-cqc98\" (UID: \"6ce9940b-e5b0-4647-bf32-532f1806d6a3\") " pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:56:30.157405 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.157378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:56:30.157506 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.157435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:56:30.159819 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.159804    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 04:56:30.160083 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.160069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 04:56:30.167824 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:30.167808    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 06 04:56:30.167902 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:30.167871    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs podName:d3a5a3f7-dc34-4f63-9fa9-5660515f8000 nodeName:}" failed. No retries permitted until 2026-09-06 04:57:34.167855068 +0000 UTC m=+130.309414987 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs") pod "network-metrics-daemon-csrcb" (UID: "d3a5a3f7-dc34-4f63-9fa9-5660515f8000") : secret "metrics-daemon-secret" not found
Sep 06 04:56:30.170024 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.170005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d08567e7-7463-408b-9d6f-d6edca57b781-original-pull-secret\") pod \"global-pull-secret-syncer-g6f2s\" (UID: \"d08567e7-7463-408b-9d6f-d6edca57b781\") " pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:56:30.253952 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.253920    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dzjkx\""
Sep 06 04:56:30.262813 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.262792    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:56:30.350367 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.350345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6f2s"
Sep 06 04:56:30.528123 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.528097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cqc98"]
Sep 06 04:56:30.531285 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:56:30.531258    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce9940b_e5b0_4647_bf32_532f1806d6a3.slice/crio-aca49090314861aade7a887f03df91910cab554d0dd794e08cbe7aeddb5a1cab WatchSource:0}: Error finding container aca49090314861aade7a887f03df91910cab554d0dd794e08cbe7aeddb5a1cab: Status 404 returned error can't find the container with id aca49090314861aade7a887f03df91910cab554d0dd794e08cbe7aeddb5a1cab
Sep 06 04:56:30.559892 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.559843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:56:30.559995 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:30.559980    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 06 04:56:30.560046 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:30.560036    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert podName:c64b1fde-c4e4-49b4-a005-7e721932b165 nodeName:}" failed. No retries permitted until 2026-09-06 04:57:02.560020697 +0000 UTC m=+98.701580635 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-dtkkv" (UID: "c64b1fde-c4e4-49b4-a005-7e721932b165") : secret "networking-console-plugin-cert" not found
Sep 06 04:56:30.611213 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.611190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g6f2s"]
Sep 06 04:56:30.613609 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:56:30.613588    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08567e7_7463_408b_9d6f_d6edca57b781.slice/crio-e06dd3d3b39f9643aaf423e5607959f117e24cdaebbdfe6ad810fdc335c2f0a0 WatchSource:0}: Error finding container e06dd3d3b39f9643aaf423e5607959f117e24cdaebbdfe6ad810fdc335c2f0a0: Status 404 returned error can't find the container with id e06dd3d3b39f9643aaf423e5607959f117e24cdaebbdfe6ad810fdc335c2f0a0
Sep 06 04:56:30.655405 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.655375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g6f2s" event={"ID":"d08567e7-7463-408b-9d6f-d6edca57b781","Type":"ContainerStarted","Data":"e06dd3d3b39f9643aaf423e5607959f117e24cdaebbdfe6ad810fdc335c2f0a0"}
Sep 06 04:56:30.656453 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.656435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cqc98" event={"ID":"6ce9940b-e5b0-4647-bf32-532f1806d6a3","Type":"ContainerStarted","Data":"aca49090314861aade7a887f03df91910cab554d0dd794e08cbe7aeddb5a1cab"}
Sep 06 04:56:30.660711 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.660689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:56:30.660802 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:30.660724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:56:30.660844 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:30.660809    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 04:56:30.660844 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:30.660818    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f689b587d-grvvf: secret "image-registry-tls" not found
Sep 06 04:56:30.660844 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:30.660829    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 04:56:30.660930 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:30.660856    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls podName:531caa8b-f2b0-41f1-9c91-b68f8fbbfa33 nodeName:}" failed. No retries permitted until 2026-09-06 04:57:02.660844814 +0000 UTC m=+98.802404733 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls") pod "image-registry-6f689b587d-grvvf" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33") : secret "image-registry-tls" not found
Sep 06 04:56:30.660930 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:56:30.660888    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls podName:84a76bc6-6f0f-4d5c-b48f-9e8ab6651556 nodeName:}" failed. No retries permitted until 2026-09-06 04:57:02.660874656 +0000 UTC m=+98.802434577 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls") pod "dns-default-mx4tj" (UID: "84a76bc6-6f0f-4d5c-b48f-9e8ab6651556") : secret "dns-default-metrics-tls" not found
Sep 06 04:56:36.673203 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:36.673147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g6f2s" event={"ID":"d08567e7-7463-408b-9d6f-d6edca57b781","Type":"ContainerStarted","Data":"719b464c49a653ee5e9bd0bc594e4ce12cc7df648e8ada35f2e186610cb90c07"}
Sep 06 04:56:36.674778 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:36.674752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cqc98" event={"ID":"6ce9940b-e5b0-4647-bf32-532f1806d6a3","Type":"ContainerStarted","Data":"3e946d6271a0daa930a967e9b91e6e8568d00c6447998bce0af32c64659fd97b"}
Sep 06 04:56:36.674896 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:36.674875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:56:36.687430 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:56:36.687385    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g6f2s" podStartSLOduration=67.65216673 podStartE2EDuration="1m12.6873683s" totalImagesPullingTime="5.03520157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:56:36.686186485 +0000 UTC m=+72.827746432" watchObservedRunningTime="2026-09-06 04:56:36.6873683 +0000 UTC m=+72.828928242"
Sep 06 04:57:02.585246 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:57:02.585211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:57:02.585676 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:57:02.585361    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 06 04:57:02.585676 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:57:02.585439    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert podName:c64b1fde-c4e4-49b4-a005-7e721932b165 nodeName:}" failed. No retries permitted until 2026-09-06 04:58:06.585422088 +0000 UTC m=+162.726982028 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-dtkkv" (UID: "c64b1fde-c4e4-49b4-a005-7e721932b165") : secret "networking-console-plugin-cert" not found
Sep 06 04:57:02.686288 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:57:02.686264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:57:02.686357 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:57:02.686341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:57:02.686425 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:57:02.686411    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 04:57:02.686463 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:57:02.686430    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f689b587d-grvvf: secret "image-registry-tls" not found
Sep 06 04:57:02.686463 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:57:02.686433    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 04:57:02.686532 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:57:02.686496    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls podName:84a76bc6-6f0f-4d5c-b48f-9e8ab6651556 nodeName:}" failed. No retries permitted until 2026-09-06 04:58:06.686479199 +0000 UTC m=+162.828039120 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls") pod "dns-default-mx4tj" (UID: "84a76bc6-6f0f-4d5c-b48f-9e8ab6651556") : secret "dns-default-metrics-tls" not found
Sep 06 04:57:02.686532 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:57:02.686511    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls podName:531caa8b-f2b0-41f1-9c91-b68f8fbbfa33 nodeName:}" failed. No retries permitted until 2026-09-06 04:58:06.686503832 +0000 UTC m=+162.828063752 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls") pod "image-registry-6f689b587d-grvvf" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33") : secret "image-registry-tls" not found
Sep 06 04:57:07.679027 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:57:07.678994    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cqc98"
Sep 06 04:57:07.693217 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:57:07.693171    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cqc98" podStartSLOduration=98.586747694 podStartE2EDuration="1m43.693137768s" totalImagesPullingTime="5.106390074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:56:36.698807453 +0000 UTC m=+72.840367421" watchObservedRunningTime="2026-09-06 04:57:07.693137768 +0000 UTC m=+103.834697712"
Sep 06 04:57:34.203252 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:57:34.203218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:57:34.203648 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:57:34.203340    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 06 04:57:34.203648 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:57:34.203391    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs podName:d3a5a3f7-dc34-4f63-9fa9-5660515f8000 nodeName:}" failed. No retries permitted until 2026-09-06 04:59:36.20337871 +0000 UTC m=+252.344938630 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs") pod "network-metrics-daemon-csrcb" (UID: "d3a5a3f7-dc34-4f63-9fa9-5660515f8000") : secret "metrics-daemon-secret" not found
Sep 06 04:58:01.755527 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:01.755486    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv" podUID="c64b1fde-c4e4-49b4-a005-7e721932b165"
Sep 06 04:58:01.769657 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:01.769622    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6f689b587d-grvvf" podUID="531caa8b-f2b0-41f1-9c91-b68f8fbbfa33"
Sep 06 04:58:01.789783 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:01.789750    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mx4tj" podUID="84a76bc6-6f0f-4d5c-b48f-9e8ab6651556"
Sep 06 04:58:01.869030 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:58:01.869007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:58:01.869185 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:58:01.869004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:58:01.869185 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:58:01.869007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:58:03.355591 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:03.355548    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 04:58:06.635701 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:58:06.635651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 04:58:06.636103 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:06.635808    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 06 04:58:06.636103 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:06.635890    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert podName:c64b1fde-c4e4-49b4-a005-7e721932b165 nodeName:}" failed. No retries permitted until 2026-09-06 05:00:08.63587437 +0000 UTC m=+284.777434294 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-dtkkv" (UID: "c64b1fde-c4e4-49b4-a005-7e721932b165") : secret "networking-console-plugin-cert" not found
Sep 06 04:58:06.736539 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:58:06.736494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 04:58:06.736539 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:58:06.736542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 04:58:06.736740 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:06.736648    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 04:58:06.736740 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:06.736650    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 04:58:06.736740 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:06.736660    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f689b587d-grvvf: secret "image-registry-tls" not found
Sep 06 04:58:06.736740 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:06.736717    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls podName:84a76bc6-6f0f-4d5c-b48f-9e8ab6651556 nodeName:}" failed. No retries permitted until 2026-09-06 05:00:08.736702062 +0000 UTC m=+284.878261982 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls") pod "dns-default-mx4tj" (UID: "84a76bc6-6f0f-4d5c-b48f-9e8ab6651556") : secret "dns-default-metrics-tls" not found
Sep 06 04:58:06.736740 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:58:06.736735    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls podName:531caa8b-f2b0-41f1-9c91-b68f8fbbfa33 nodeName:}" failed. No retries permitted until 2026-09-06 05:00:08.736725671 +0000 UTC m=+284.878285591 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls") pod "image-registry-6f689b587d-grvvf" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33") : secret "image-registry-tls" not found
Sep 06 04:58:15.341898 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:58:15.341847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:59:36.235824 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:36.235776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 04:59:36.236332 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:36.235920    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 06 04:59:36.236332 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:36.235995    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs podName:d3a5a3f7-dc34-4f63-9fa9-5660515f8000 nodeName:}" failed. No retries permitted until 2026-09-06 05:01:38.235980516 +0000 UTC m=+374.377540436 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs") pod "network-metrics-daemon-csrcb" (UID: "d3a5a3f7-dc34-4f63-9fa9-5660515f8000") : secret "metrics-daemon-secret" not found
Sep 06 04:59:41.382718 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.382679    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"]
Sep 06 04:59:41.384781 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.384762    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"]
Sep 06 04:59:41.384925 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.384892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.386748 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.386726    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"]
Sep 06 04:59:41.386849 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.386823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:41.387323 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.387302    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 06 04:59:41.387436 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.387308    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 06 04:59:41.388060 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.388045    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hxp8l\""
Sep 06 04:59:41.388381 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.388367    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 06 04:59:41.388432 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.388406    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 06 04:59:41.389001 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.388985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.389442 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.389424    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 06 04:59:41.389535 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.389426    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sjnzb\""
Sep 06 04:59:41.389677 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.389662    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 06 04:59:41.389962 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.389945    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 06 04:59:41.390981 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.390964    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 06 04:59:41.391073 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.390980    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 06 04:59:41.391350 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.391332    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 06 04:59:41.391440 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.391378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pbqkg\""
Sep 06 04:59:41.391949 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.391786    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 06 04:59:41.399613 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.399566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"]
Sep 06 04:59:41.400266 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.400251    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"]
Sep 06 04:59:41.402466 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.402447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"]
Sep 06 04:59:41.476500 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.476465    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-gkf6c"]
Sep 06 04:59:41.478308 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.478288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-gkf6c"
Sep 06 04:59:41.480459 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.480440    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-b4mjt\""
Sep 06 04:59:41.486411 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.486392    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-gkf6c"]
Sep 06 04:59:41.570404 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.570378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:41.570504 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.570406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1f7670f-7713-48fc-acaa-81b99459a98b-config\") pod \"service-ca-operator-859f94c4b-pnlgs\" (UID: \"c1f7670f-7713-48fc-acaa-81b99459a98b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.570504 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.570427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drnjn\" (UniqueName: \"kubernetes.io/projected/c1f7670f-7713-48fc-acaa-81b99459a98b-kube-api-access-drnjn\") pod \"service-ca-operator-859f94c4b-pnlgs\" (UID: \"c1f7670f-7713-48fc-acaa-81b99459a98b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.570504 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.570443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97rz\" (UniqueName: \"kubernetes.io/projected/a21526c5-2c0d-48dc-89c1-11b2abd6dac3-kube-api-access-f97rz\") pod \"kube-storage-version-migrator-operator-8d7dd4675-x6wn5\" (UID: \"a21526c5-2c0d-48dc-89c1-11b2abd6dac3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.570504 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.570471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a21526c5-2c0d-48dc-89c1-11b2abd6dac3-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-x6wn5\" (UID: \"a21526c5-2c0d-48dc-89c1-11b2abd6dac3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.570634 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.570561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4wp\" (UniqueName: \"kubernetes.io/projected/9e96d3f3-4155-48e3-bf65-db92a5873fe6-kube-api-access-kk4wp\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:41.570634 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.570585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1f7670f-7713-48fc-acaa-81b99459a98b-serving-cert\") pod \"service-ca-operator-859f94c4b-pnlgs\" (UID: \"c1f7670f-7713-48fc-acaa-81b99459a98b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.570634 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.570605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a21526c5-2c0d-48dc-89c1-11b2abd6dac3-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-x6wn5\" (UID: \"a21526c5-2c0d-48dc-89c1-11b2abd6dac3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.570634 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.570627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6bwm\" (UniqueName: \"kubernetes.io/projected/8d0ce0a2-f863-4a69-ba4b-729c6471e551-kube-api-access-g6bwm\") pod \"network-check-source-685456655f-gkf6c\" (UID: \"8d0ce0a2-f863-4a69-ba4b-729c6471e551\") " pod="openshift-network-diagnostics/network-check-source-685456655f-gkf6c"
Sep 06 04:59:41.671782 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.671718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:41.671782 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.671746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1f7670f-7713-48fc-acaa-81b99459a98b-config\") pod \"service-ca-operator-859f94c4b-pnlgs\" (UID: \"c1f7670f-7713-48fc-acaa-81b99459a98b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.671782 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.671766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drnjn\" (UniqueName: \"kubernetes.io/projected/c1f7670f-7713-48fc-acaa-81b99459a98b-kube-api-access-drnjn\") pod \"service-ca-operator-859f94c4b-pnlgs\" (UID: \"c1f7670f-7713-48fc-acaa-81b99459a98b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.671981 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.671797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97rz\" (UniqueName: \"kubernetes.io/projected/a21526c5-2c0d-48dc-89c1-11b2abd6dac3-kube-api-access-f97rz\") pod \"kube-storage-version-migrator-operator-8d7dd4675-x6wn5\" (UID: \"a21526c5-2c0d-48dc-89c1-11b2abd6dac3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.671981 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.671828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a21526c5-2c0d-48dc-89c1-11b2abd6dac3-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-x6wn5\" (UID: \"a21526c5-2c0d-48dc-89c1-11b2abd6dac3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.671981 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:41.671869    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 04:59:41.671981 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.671887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4wp\" (UniqueName: \"kubernetes.io/projected/9e96d3f3-4155-48e3-bf65-db92a5873fe6-kube-api-access-kk4wp\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:41.671981 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.671915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1f7670f-7713-48fc-acaa-81b99459a98b-serving-cert\") pod \"service-ca-operator-859f94c4b-pnlgs\" (UID: \"c1f7670f-7713-48fc-acaa-81b99459a98b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.671981 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:41.671948    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls podName:9e96d3f3-4155-48e3-bf65-db92a5873fe6 nodeName:}" failed. No retries permitted until 2026-09-06 04:59:42.171929008 +0000 UTC m=+258.313488947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-ccf9h" (UID: "9e96d3f3-4155-48e3-bf65-db92a5873fe6") : secret "samples-operator-tls" not found
Sep 06 04:59:41.671981 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.671970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a21526c5-2c0d-48dc-89c1-11b2abd6dac3-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-x6wn5\" (UID: \"a21526c5-2c0d-48dc-89c1-11b2abd6dac3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.672356 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.672000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6bwm\" (UniqueName: \"kubernetes.io/projected/8d0ce0a2-f863-4a69-ba4b-729c6471e551-kube-api-access-g6bwm\") pod \"network-check-source-685456655f-gkf6c\" (UID: \"8d0ce0a2-f863-4a69-ba4b-729c6471e551\") " pod="openshift-network-diagnostics/network-check-source-685456655f-gkf6c"
Sep 06 04:59:41.672460 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.672439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1f7670f-7713-48fc-acaa-81b99459a98b-config\") pod \"service-ca-operator-859f94c4b-pnlgs\" (UID: \"c1f7670f-7713-48fc-acaa-81b99459a98b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.672565 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.672544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a21526c5-2c0d-48dc-89c1-11b2abd6dac3-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-x6wn5\" (UID: \"a21526c5-2c0d-48dc-89c1-11b2abd6dac3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.674278 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.674261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1f7670f-7713-48fc-acaa-81b99459a98b-serving-cert\") pod \"service-ca-operator-859f94c4b-pnlgs\" (UID: \"c1f7670f-7713-48fc-acaa-81b99459a98b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.674278 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.674274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a21526c5-2c0d-48dc-89c1-11b2abd6dac3-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-x6wn5\" (UID: \"a21526c5-2c0d-48dc-89c1-11b2abd6dac3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.680570 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.680550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drnjn\" (UniqueName: \"kubernetes.io/projected/c1f7670f-7713-48fc-acaa-81b99459a98b-kube-api-access-drnjn\") pod \"service-ca-operator-859f94c4b-pnlgs\" (UID: \"c1f7670f-7713-48fc-acaa-81b99459a98b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.680713 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.680694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97rz\" (UniqueName: \"kubernetes.io/projected/a21526c5-2c0d-48dc-89c1-11b2abd6dac3-kube-api-access-f97rz\") pod \"kube-storage-version-migrator-operator-8d7dd4675-x6wn5\" (UID: \"a21526c5-2c0d-48dc-89c1-11b2abd6dac3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.681370 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.681355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6bwm\" (UniqueName: \"kubernetes.io/projected/8d0ce0a2-f863-4a69-ba4b-729c6471e551-kube-api-access-g6bwm\") pod \"network-check-source-685456655f-gkf6c\" (UID: \"8d0ce0a2-f863-4a69-ba4b-729c6471e551\") " pod="openshift-network-diagnostics/network-check-source-685456655f-gkf6c"
Sep 06 04:59:41.681744 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.681726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4wp\" (UniqueName: \"kubernetes.io/projected/9e96d3f3-4155-48e3-bf65-db92a5873fe6-kube-api-access-kk4wp\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:41.695239 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.695222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"
Sep 06 04:59:41.707622 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.707604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"
Sep 06 04:59:41.786693 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.786669    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-gkf6c"
Sep 06 04:59:41.965041 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.965010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs"]
Sep 06 04:59:41.968256 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:59:41.968225    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f7670f_7713_48fc_acaa_81b99459a98b.slice/crio-c3318ca36f70aa0b3ec355b89a2efa54cc062adda4c2f206bd62b8213c529739 WatchSource:0}: Error finding container c3318ca36f70aa0b3ec355b89a2efa54cc062adda4c2f206bd62b8213c529739: Status 404 returned error can't find the container with id c3318ca36f70aa0b3ec355b89a2efa54cc062adda4c2f206bd62b8213c529739
Sep 06 04:59:41.981619 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:41.981594    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5"]
Sep 06 04:59:41.984439 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:59:41.984416    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21526c5_2c0d_48dc_89c1_11b2abd6dac3.slice/crio-5faac2b90d88be6cbd6480b452706d18fe4ed656f359117df0ed0a559e99f564 WatchSource:0}: Error finding container 5faac2b90d88be6cbd6480b452706d18fe4ed656f359117df0ed0a559e99f564: Status 404 returned error can't find the container with id 5faac2b90d88be6cbd6480b452706d18fe4ed656f359117df0ed0a559e99f564
Sep 06 04:59:42.049099 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:42.049075    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-gkf6c"]
Sep 06 04:59:42.051494 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:59:42.051470    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0ce0a2_f863_4a69_ba4b_729c6471e551.slice/crio-ffbf0ed00fa4d471388b9b814d2404b16f06a54f18a0d5e6cef1d8884b37b85a WatchSource:0}: Error finding container ffbf0ed00fa4d471388b9b814d2404b16f06a54f18a0d5e6cef1d8884b37b85a: Status 404 returned error can't find the container with id ffbf0ed00fa4d471388b9b814d2404b16f06a54f18a0d5e6cef1d8884b37b85a
Sep 06 04:59:42.097244 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:42.097214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-gkf6c" event={"ID":"8d0ce0a2-f863-4a69-ba4b-729c6471e551","Type":"ContainerStarted","Data":"ffbf0ed00fa4d471388b9b814d2404b16f06a54f18a0d5e6cef1d8884b37b85a"}
Sep 06 04:59:42.098402 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:42.098380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5" event={"ID":"a21526c5-2c0d-48dc-89c1-11b2abd6dac3","Type":"ContainerStarted","Data":"5faac2b90d88be6cbd6480b452706d18fe4ed656f359117df0ed0a559e99f564"}
Sep 06 04:59:42.099554 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:42.099524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs" event={"ID":"c1f7670f-7713-48fc-acaa-81b99459a98b","Type":"ContainerStarted","Data":"c3318ca36f70aa0b3ec355b89a2efa54cc062adda4c2f206bd62b8213c529739"}
Sep 06 04:59:42.176246 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:42.176208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:42.176404 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:42.176362    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 04:59:42.176471 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:42.176421    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls podName:9e96d3f3-4155-48e3-bf65-db92a5873fe6 nodeName:}" failed. No retries permitted until 2026-09-06 04:59:43.176404293 +0000 UTC m=+259.317964214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-ccf9h" (UID: "9e96d3f3-4155-48e3-bf65-db92a5873fe6") : secret "samples-operator-tls" not found
Sep 06 04:59:43.103946 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:43.103910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-gkf6c" event={"ID":"8d0ce0a2-f863-4a69-ba4b-729c6471e551","Type":"ContainerStarted","Data":"369de6df7380d87de7f078fee6d74245df195d2e47ee8aa9cb0a506ba3b8e545"}
Sep 06 04:59:43.119770 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:43.119696    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-gkf6c" podStartSLOduration=2.11967556 podStartE2EDuration="2.11967556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:59:43.118359967 +0000 UTC m=+259.259919933" watchObservedRunningTime="2026-09-06 04:59:43.11967556 +0000 UTC m=+259.261235504"
Sep 06 04:59:43.185856 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:43.185824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:43.186013 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:43.185974    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 04:59:43.186059 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:43.186043    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls podName:9e96d3f3-4155-48e3-bf65-db92a5873fe6 nodeName:}" failed. No retries permitted until 2026-09-06 04:59:45.186029415 +0000 UTC m=+261.327589336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-ccf9h" (UID: "9e96d3f3-4155-48e3-bf65-db92a5873fe6") : secret "samples-operator-tls" not found
Sep 06 04:59:45.111670 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.111630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5" event={"ID":"a21526c5-2c0d-48dc-89c1-11b2abd6dac3","Type":"ContainerStarted","Data":"687e0eb5a0852d41ec84b66c7bfc760c651df87eea2306d17f5e3c46eb90c7cc"}
Sep 06 04:59:45.113077 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.113052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs" event={"ID":"c1f7670f-7713-48fc-acaa-81b99459a98b","Type":"ContainerStarted","Data":"22aa7ab0063008b709d3421fb534221037de9683791ef5dd094fffbbb60f6dd4"}
Sep 06 04:59:45.126354 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.126310    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5" podStartSLOduration=1.841303055 podStartE2EDuration="4.126296622s" totalImagesPullingTime="2.284993567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:59:45.125833891 +0000 UTC m=+261.267393855" watchObservedRunningTime="2026-09-06 04:59:45.126296622 +0000 UTC m=+261.267856564"
Sep 06 04:59:45.203273 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.203242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:45.203426 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:45.203391    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 04:59:45.203469 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:45.203456    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls podName:9e96d3f3-4155-48e3-bf65-db92a5873fe6 nodeName:}" failed. No retries permitted until 2026-09-06 04:59:49.203440311 +0000 UTC m=+265.345000232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-ccf9h" (UID: "9e96d3f3-4155-48e3-bf65-db92a5873fe6") : secret "samples-operator-tls" not found
Sep 06 04:59:45.753503 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.753441    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs" podStartSLOduration=2.450451402 podStartE2EDuration="4.75341783s" totalImagesPullingTime="2.302966428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:59:45.138793937 +0000 UTC m=+261.280353891" watchObservedRunningTime="2026-09-06 04:59:45.75341783 +0000 UTC m=+261.894977774"
Sep 06 04:59:45.754310 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.754283    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rq4r9"]
Sep 06 04:59:45.757426 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.757407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:45.759729 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.759704    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 06 04:59:45.759729 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.759718    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 06 04:59:45.759883 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.759713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qr8nl\""
Sep 06 04:59:45.760507 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.760493    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 06 04:59:45.760552 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.760514    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 06 04:59:45.768062 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.768038    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rq4r9"]
Sep 06 04:59:45.907913 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.907885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f211ba82-097b-48c4-8449-a43018d3bcda-data-volume\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:45.908093 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.907918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:45.908093 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.908015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:45.908245 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.908105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfcs\" (UniqueName: \"kubernetes.io/projected/f211ba82-097b-48c4-8449-a43018d3bcda-kube-api-access-cqfcs\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:45.908245 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:45.908197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f211ba82-097b-48c4-8449-a43018d3bcda-crio-socket\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.008690 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.008612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.008690 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.008659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfcs\" (UniqueName: \"kubernetes.io/projected/f211ba82-097b-48c4-8449-a43018d3bcda-kube-api-access-cqfcs\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.008690 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.008687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f211ba82-097b-48c4-8449-a43018d3bcda-crio-socket\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.008901 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.008706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f211ba82-097b-48c4-8449-a43018d3bcda-data-volume\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.008901 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.008720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.008901 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:46.008764    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 04:59:46.008901 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:46.008786    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 04:59:46.008901 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.008804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f211ba82-097b-48c4-8449-a43018d3bcda-crio-socket\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.008901 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:46.008842    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls podName:f211ba82-097b-48c4-8449-a43018d3bcda nodeName:}" failed. No retries permitted until 2026-09-06 04:59:46.50882323 +0000 UTC m=+262.650383151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq4r9" (UID: "f211ba82-097b-48c4-8449-a43018d3bcda") : secret "insights-runtime-extractor-tls" not found
Sep 06 04:59:46.008901 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:46.008864    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm podName:f211ba82-097b-48c4-8449-a43018d3bcda nodeName:}" failed. No retries permitted until 2026-09-06 04:59:46.508857014 +0000 UTC m=+262.650416933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq4r9" (UID: "f211ba82-097b-48c4-8449-a43018d3bcda") : configmap "kube-rbac-proxy" not found
Sep 06 04:59:46.009150 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.009008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f211ba82-097b-48c4-8449-a43018d3bcda-data-volume\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.019654 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.019631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfcs\" (UniqueName: \"kubernetes.io/projected/f211ba82-097b-48c4-8449-a43018d3bcda-kube-api-access-cqfcs\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.443193 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.443099    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm"]
Sep 06 04:59:46.446079 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.446061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm"
Sep 06 04:59:46.448214 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.448191    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d9rdq\""
Sep 06 04:59:46.448408 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.448381    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 06 04:59:46.449102 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.449080    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 06 04:59:46.455432 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.455411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm"]
Sep 06 04:59:46.514119 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.514083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.514264 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:46.514208    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 04:59:46.514264 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.514212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:46.514264 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:46.514260    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm podName:f211ba82-097b-48c4-8449-a43018d3bcda nodeName:}" failed. No retries permitted until 2026-09-06 04:59:47.514244504 +0000 UTC m=+263.655804424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq4r9" (UID: "f211ba82-097b-48c4-8449-a43018d3bcda") : configmap "kube-rbac-proxy" not found
Sep 06 04:59:46.514444 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.514301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5nf\" (UniqueName: \"kubernetes.io/projected/210495ce-119c-4189-8b5d-72a22d362b66-kube-api-access-xg5nf\") pod \"migrator-fdbb8fdff-5kpkm\" (UID: \"210495ce-119c-4189-8b5d-72a22d362b66\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm"
Sep 06 04:59:46.514444 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:46.514336    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 04:59:46.514444 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:46.514384    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls podName:f211ba82-097b-48c4-8449-a43018d3bcda nodeName:}" failed. No retries permitted until 2026-09-06 04:59:47.514368365 +0000 UTC m=+263.655928292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq4r9" (UID: "f211ba82-097b-48c4-8449-a43018d3bcda") : secret "insights-runtime-extractor-tls" not found
Sep 06 04:59:46.614731 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.614685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5nf\" (UniqueName: \"kubernetes.io/projected/210495ce-119c-4189-8b5d-72a22d362b66-kube-api-access-xg5nf\") pod \"migrator-fdbb8fdff-5kpkm\" (UID: \"210495ce-119c-4189-8b5d-72a22d362b66\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm"
Sep 06 04:59:46.622766 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.622740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5nf\" (UniqueName: \"kubernetes.io/projected/210495ce-119c-4189-8b5d-72a22d362b66-kube-api-access-xg5nf\") pod \"migrator-fdbb8fdff-5kpkm\" (UID: \"210495ce-119c-4189-8b5d-72a22d362b66\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm"
Sep 06 04:59:46.756773 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:46.756744    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm"
Sep 06 04:59:47.022954 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.022878    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm"]
Sep 06 04:59:47.025799 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:59:47.025774    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210495ce_119c_4189_8b5d_72a22d362b66.slice/crio-ccc658a71d0328308912e0dca33111738a785b13a8db04e940a936f6b7547817 WatchSource:0}: Error finding container ccc658a71d0328308912e0dca33111738a785b13a8db04e940a936f6b7547817: Status 404 returned error can't find the container with id ccc658a71d0328308912e0dca33111738a785b13a8db04e940a936f6b7547817
Sep 06 04:59:47.120776 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.120737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm" event={"ID":"210495ce-119c-4189-8b5d-72a22d362b66","Type":"ContainerStarted","Data":"ccc658a71d0328308912e0dca33111738a785b13a8db04e940a936f6b7547817"}
Sep 06 04:59:47.484600 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.484568    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-8clvz"]
Sep 06 04:59:47.488636 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.488622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.491100 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.491070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 06 04:59:47.491100 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.491087    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 06 04:59:47.491306 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.491182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 06 04:59:47.491306 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.491181    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 06 04:59:47.492001 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.491984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-h4cxv\""
Sep 06 04:59:47.494461 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.494439    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-8clvz"]
Sep 06 04:59:47.522227 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.522201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsgrp\" (UniqueName: \"kubernetes.io/projected/063fc7a9-0fbe-4ffa-9481-cb2c9592cc61-kube-api-access-tsgrp\") pod \"service-ca-75d99c484-8clvz\" (UID: \"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61\") " pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.522345 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.522231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:47.522345 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.522300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/063fc7a9-0fbe-4ffa-9481-cb2c9592cc61-signing-cabundle\") pod \"service-ca-75d99c484-8clvz\" (UID: \"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61\") " pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.522345 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:47.522327    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 04:59:47.522345 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.522341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:47.522550 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:47.522367    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls podName:f211ba82-097b-48c4-8449-a43018d3bcda nodeName:}" failed. No retries permitted until 2026-09-06 04:59:49.522355439 +0000 UTC m=+265.663915359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq4r9" (UID: "f211ba82-097b-48c4-8449-a43018d3bcda") : secret "insights-runtime-extractor-tls" not found
Sep 06 04:59:47.522550 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:47.522411    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 04:59:47.522550 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.522416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/063fc7a9-0fbe-4ffa-9481-cb2c9592cc61-signing-key\") pod \"service-ca-75d99c484-8clvz\" (UID: \"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61\") " pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.522550 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:47.522472    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm podName:f211ba82-097b-48c4-8449-a43018d3bcda nodeName:}" failed. No retries permitted until 2026-09-06 04:59:49.522456329 +0000 UTC m=+265.664016264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq4r9" (UID: "f211ba82-097b-48c4-8449-a43018d3bcda") : configmap "kube-rbac-proxy" not found
Sep 06 04:59:47.623119 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.623083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/063fc7a9-0fbe-4ffa-9481-cb2c9592cc61-signing-cabundle\") pod \"service-ca-75d99c484-8clvz\" (UID: \"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61\") " pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.623316 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.623146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/063fc7a9-0fbe-4ffa-9481-cb2c9592cc61-signing-key\") pod \"service-ca-75d99c484-8clvz\" (UID: \"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61\") " pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.623316 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.623228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsgrp\" (UniqueName: \"kubernetes.io/projected/063fc7a9-0fbe-4ffa-9481-cb2c9592cc61-kube-api-access-tsgrp\") pod \"service-ca-75d99c484-8clvz\" (UID: \"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61\") " pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.623920 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.623897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/063fc7a9-0fbe-4ffa-9481-cb2c9592cc61-signing-cabundle\") pod \"service-ca-75d99c484-8clvz\" (UID: \"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61\") " pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.626103 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.626070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/063fc7a9-0fbe-4ffa-9481-cb2c9592cc61-signing-key\") pod \"service-ca-75d99c484-8clvz\" (UID: \"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61\") " pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.630931 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.630911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsgrp\" (UniqueName: \"kubernetes.io/projected/063fc7a9-0fbe-4ffa-9481-cb2c9592cc61-kube-api-access-tsgrp\") pod \"service-ca-75d99c484-8clvz\" (UID: \"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61\") " pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.799130 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.799048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-8clvz"
Sep 06 04:59:47.931468 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:47.931437    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kwqp6_fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96/dns-node-resolver/0.log"
Sep 06 04:59:48.285180 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:48.285131    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-8clvz"]
Sep 06 04:59:48.287735 ip-10-0-128-168 kubenswrapper[2580]: W0906 04:59:48.287706    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063fc7a9_0fbe_4ffa_9481_cb2c9592cc61.slice/crio-d62d5a660e495d4ae40b6bee687f476d162622c8bf8421fc12f2f135bb69ff98 WatchSource:0}: Error finding container d62d5a660e495d4ae40b6bee687f476d162622c8bf8421fc12f2f135bb69ff98: Status 404 returned error can't find the container with id d62d5a660e495d4ae40b6bee687f476d162622c8bf8421fc12f2f135bb69ff98
Sep 06 04:59:48.731239 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:48.731215    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hr7nb_53aa3ac4-3e77-40df-9136-ff0e41c2606b/node-ca/0.log"
Sep 06 04:59:49.131199 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:49.131081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-8clvz" event={"ID":"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61","Type":"ContainerStarted","Data":"f505f23330b264582fe949281419aa0c28d3cf4d0f49cc7e411427fc61fb5dc3"}
Sep 06 04:59:49.131199 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:49.131141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-8clvz" event={"ID":"063fc7a9-0fbe-4ffa-9481-cb2c9592cc61","Type":"ContainerStarted","Data":"d62d5a660e495d4ae40b6bee687f476d162622c8bf8421fc12f2f135bb69ff98"}
Sep 06 04:59:49.133109 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:49.133085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm" event={"ID":"210495ce-119c-4189-8b5d-72a22d362b66","Type":"ContainerStarted","Data":"0845b8367f4f045bbac5cccab3b0c814f249d8658f654f8f926a37cf0a249e70"}
Sep 06 04:59:49.133318 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:49.133126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm" event={"ID":"210495ce-119c-4189-8b5d-72a22d362b66","Type":"ContainerStarted","Data":"989d5740078c6f37e7b13eefd816b8cb8203e0ba64f644857f5f0efe27e4ac78"}
Sep 06 04:59:49.145753 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:49.145707    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-8clvz" podStartSLOduration=2.145694425 podStartE2EDuration="2.145694425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:59:49.14448966 +0000 UTC m=+265.286049604" watchObservedRunningTime="2026-09-06 04:59:49.145694425 +0000 UTC m=+265.287254660"
Sep 06 04:59:49.157869 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:49.157826    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5kpkm" podStartSLOduration=2.113211766 podStartE2EDuration="3.1578083s" totalImagesPullingTime="1.044596534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 04:59:49.1571881 +0000 UTC m=+265.298748056" watchObservedRunningTime="2026-09-06 04:59:49.1578083 +0000 UTC m=+265.299368243"
Sep 06 04:59:49.236911 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:49.236865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:49.237058 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:49.236962    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 04:59:49.237058 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:49.237033    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls podName:9e96d3f3-4155-48e3-bf65-db92a5873fe6 nodeName:}" failed. No retries permitted until 2026-09-06 04:59:57.237016079 +0000 UTC m=+273.378576014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-ccf9h" (UID: "9e96d3f3-4155-48e3-bf65-db92a5873fe6") : secret "samples-operator-tls" not found
Sep 06 04:59:49.331747 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:49.331657    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-55l5h_8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3/serve-healthcheck-canary/0.log"
Sep 06 04:59:49.539310 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:49.539236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:49.539524 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:49.539343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:49.539524 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:49.539398    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 04:59:49.539524 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:49.539463    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 04:59:49.539524 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:49.539471    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls podName:f211ba82-097b-48c4-8449-a43018d3bcda nodeName:}" failed. No retries permitted until 2026-09-06 04:59:53.539449846 +0000 UTC m=+269.681009781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq4r9" (UID: "f211ba82-097b-48c4-8449-a43018d3bcda") : secret "insights-runtime-extractor-tls" not found
Sep 06 04:59:49.539524 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:49.539523    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm podName:f211ba82-097b-48c4-8449-a43018d3bcda nodeName:}" failed. No retries permitted until 2026-09-06 04:59:53.539507795 +0000 UTC m=+269.681067720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq4r9" (UID: "f211ba82-097b-48c4-8449-a43018d3bcda") : configmap "kube-rbac-proxy" not found
Sep 06 04:59:53.570975 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:53.570943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:53.571365 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:53.571010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 04:59:53.571365 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:53.571071    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 04:59:53.571365 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:53.571087    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 04:59:53.571365 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:53.571125    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm podName:f211ba82-097b-48c4-8449-a43018d3bcda nodeName:}" failed. No retries permitted until 2026-09-06 05:00:01.57111159 +0000 UTC m=+277.712671510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq4r9" (UID: "f211ba82-097b-48c4-8449-a43018d3bcda") : configmap "kube-rbac-proxy" not found
Sep 06 04:59:53.571365 ip-10-0-128-168 kubenswrapper[2580]: E0906 04:59:53.571138    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls podName:f211ba82-097b-48c4-8449-a43018d3bcda nodeName:}" failed. No retries permitted until 2026-09-06 05:00:01.571132563 +0000 UTC m=+277.712692483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq4r9" (UID: "f211ba82-097b-48c4-8449-a43018d3bcda") : secret "insights-runtime-extractor-tls" not found
Sep 06 04:59:57.300669 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:57.300633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:57.303105 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:57.303073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9e96d3f3-4155-48e3-bf65-db92a5873fe6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-ccf9h\" (UID: \"9e96d3f3-4155-48e3-bf65-db92a5873fe6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:57.602226 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:57.602118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"
Sep 06 04:59:57.871916 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:57.871840    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h"]
Sep 06 04:59:58.161799 ip-10-0-128-168 kubenswrapper[2580]: I0906 04:59:58.161713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h" event={"ID":"9e96d3f3-4155-48e3-bf65-db92a5873fe6","Type":"ContainerStarted","Data":"9a31f86afed2f195eb9394bacfd37af48db8d72a66c83f3aa7dec253a5a1600e"}
Sep 06 05:00:00.169740 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:00.169706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h" event={"ID":"9e96d3f3-4155-48e3-bf65-db92a5873fe6","Type":"ContainerStarted","Data":"57e57003635a33146271e79fea2ba8d27878560da7d63ebd26f28eecff21ed07"}
Sep 06 05:00:00.170202 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:00.169761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h" event={"ID":"9e96d3f3-4155-48e3-bf65-db92a5873fe6","Type":"ContainerStarted","Data":"e751e38e081999547de7a1634ad5306d21f8d8a12ed655e0a359daec29df08f5"}
Sep 06 05:00:00.186137 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:00.186084    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-ccf9h" podStartSLOduration=17.642936195 podStartE2EDuration="19.186069358s" totalImagesPullingTime="1.543133163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:00.18465758 +0000 UTC m=+276.326217545" watchObservedRunningTime="2026-09-06 05:00:00.186069358 +0000 UTC m=+276.327629300"
Sep 06 05:00:01.634330 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:01.634292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 05:00:01.634659 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:01.634369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 05:00:01.634802 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:01.634782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211ba82-097b-48c4-8449-a43018d3bcda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 05:00:01.636629 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:01.636613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211ba82-097b-48c4-8449-a43018d3bcda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq4r9\" (UID: \"f211ba82-097b-48c4-8449-a43018d3bcda\") " pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 05:00:01.666656 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:01.666630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rq4r9"
Sep 06 05:00:01.933951 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:01.933832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rq4r9"]
Sep 06 05:00:01.936374 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:01.936347    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf211ba82_097b_48c4_8449_a43018d3bcda.slice/crio-68d32d1bdf21a3d087444a35c80c8b5a4436d67341b287d732e387ecd0d3b485 WatchSource:0}: Error finding container 68d32d1bdf21a3d087444a35c80c8b5a4436d67341b287d732e387ecd0d3b485: Status 404 returned error can't find the container with id 68d32d1bdf21a3d087444a35c80c8b5a4436d67341b287d732e387ecd0d3b485
Sep 06 05:00:02.177770 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:02.177737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq4r9" event={"ID":"f211ba82-097b-48c4-8449-a43018d3bcda","Type":"ContainerStarted","Data":"f91b1e3a2f527647bc01feca3929b4e2b75849af8eb30ceb3dc34381161a6a91"}
Sep 06 05:00:02.177908 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:02.177788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq4r9" event={"ID":"f211ba82-097b-48c4-8449-a43018d3bcda","Type":"ContainerStarted","Data":"68d32d1bdf21a3d087444a35c80c8b5a4436d67341b287d732e387ecd0d3b485"}
Sep 06 05:00:03.183002 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:03.182955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq4r9" event={"ID":"f211ba82-097b-48c4-8449-a43018d3bcda","Type":"ContainerStarted","Data":"f45a07ebfffe052e3f6ff40d82098f30705b7766f62033a8b980c08e31db92ca"}
Sep 06 05:00:04.870332 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:04.870291    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6f689b587d-grvvf" podUID="531caa8b-f2b0-41f1-9c91-b68f8fbbfa33"
Sep 06 05:00:04.870332 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:04.870291    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv" podUID="c64b1fde-c4e4-49b4-a005-7e721932b165"
Sep 06 05:00:04.870332 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:04.870291    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mx4tj" podUID="84a76bc6-6f0f-4d5c-b48f-9e8ab6651556"
Sep 06 05:00:05.191010 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:05.190949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mx4tj"
Sep 06 05:00:05.191010 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:05.190988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 05:00:05.191010 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:05.190984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq4r9" event={"ID":"f211ba82-097b-48c4-8449-a43018d3bcda","Type":"ContainerStarted","Data":"e68af8e4d740fedd990807fa1fd3eef9879a9b2d0141726d5a2689a598edb688"}
Sep 06 05:00:05.191243 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:05.191133    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 05:00:05.206963 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:05.206922    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rq4r9" podStartSLOduration=17.240321006 podStartE2EDuration="20.206910468s" totalImagesPullingTime="2.966589462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 04:59:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:05.2058965 +0000 UTC m=+281.347456464" watchObservedRunningTime="2026-09-06 05:00:05.206910468 +0000 UTC m=+281.348470410"
Sep 06 05:00:08.691282 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:08.691248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 05:00:08.693716 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:08.693692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c64b1fde-c4e4-49b4-a005-7e721932b165-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-dtkkv\" (UID: \"c64b1fde-c4e4-49b4-a005-7e721932b165\") " pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 05:00:08.791957 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:08.791922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 05:00:08.792122 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:08.791986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 05:00:08.794000 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:08.793973    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wzm9q\""
Sep 06 05:00:08.794431 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:08.794403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a76bc6-6f0f-4d5c-b48f-9e8ab6651556-metrics-tls\") pod \"dns-default-mx4tj\" (UID: \"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556\") " pod="openshift-dns/dns-default-mx4tj"
Sep 06 05:00:08.794554 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:08.794538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"image-registry-6f689b587d-grvvf\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") " pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 05:00:08.802723 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:08.802706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv"
Sep 06 05:00:09.070472 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.070438    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-dtkkv"]
Sep 06 05:00:09.073582 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:09.073559    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64b1fde_c4e4_49b4_a005_7e721932b165.slice/crio-9389d7f9ab3b4d9342854b6afe9a0283db9eb992a9f75fa14aeae9cd5307383b WatchSource:0}: Error finding container 9389d7f9ab3b4d9342854b6afe9a0283db9eb992a9f75fa14aeae9cd5307383b: Status 404 returned error can't find the container with id 9389d7f9ab3b4d9342854b6afe9a0283db9eb992a9f75fa14aeae9cd5307383b
Sep 06 05:00:09.094559 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.094536    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fgl4q\""
Sep 06 05:00:09.094663 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.094537    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-prh2m\""
Sep 06 05:00:09.102422 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.102399    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 05:00:09.102485 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.102473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mx4tj"
Sep 06 05:00:09.204298 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.204266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv" event={"ID":"c64b1fde-c4e4-49b4-a005-7e721932b165","Type":"ContainerStarted","Data":"9389d7f9ab3b4d9342854b6afe9a0283db9eb992a9f75fa14aeae9cd5307383b"}
Sep 06 05:00:09.382980 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.382915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mx4tj"]
Sep 06 05:00:09.386321 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.386297    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f689b587d-grvvf"]
Sep 06 05:00:09.386480 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:09.386461    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a76bc6_6f0f_4d5c_b48f_9e8ab6651556.slice/crio-4a7990bfd78af96e58c3e3594212317fd36e90d0335ff434e1263abf28ee8b8f WatchSource:0}: Error finding container 4a7990bfd78af96e58c3e3594212317fd36e90d0335ff434e1263abf28ee8b8f: Status 404 returned error can't find the container with id 4a7990bfd78af96e58c3e3594212317fd36e90d0335ff434e1263abf28ee8b8f
Sep 06 05:00:09.389528 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:09.389503    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531caa8b_f2b0_41f1_9c91_b68f8fbbfa33.slice/crio-bbb189692de10227c51b1afa597fd9ec353ada51ef92ee21c3fcd92f5c717401 WatchSource:0}: Error finding container bbb189692de10227c51b1afa597fd9ec353ada51ef92ee21c3fcd92f5c717401: Status 404 returned error can't find the container with id bbb189692de10227c51b1afa597fd9ec353ada51ef92ee21c3fcd92f5c717401
Sep 06 05:00:09.704830 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.704474    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"]
Sep 06 05:00:09.707948 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.707919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.711386 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.711145    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 06 05:00:09.711386 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.711174    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 06 05:00:09.712018 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.711999    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 06 05:00:09.712130 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.712088    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 06 05:00:09.712130 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.712114    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rxp5m\""
Sep 06 05:00:09.712363 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.712218    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 06 05:00:09.729562 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.729532    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"]
Sep 06 05:00:09.800209 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.800176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.800402 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.800235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.800402 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.800314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.800402 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.800363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm95t\" (UniqueName: \"kubernetes.io/projected/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-kube-api-access-vm95t\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.901109 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.901073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.901309 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.901128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.901309 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.901194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.901309 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.901225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm95t\" (UniqueName: \"kubernetes.io/projected/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-kube-api-access-vm95t\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.901858 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.901835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.904039 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.904008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.904194 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.904172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:09.908558 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:09.908539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm95t\" (UniqueName: \"kubernetes.io/projected/6dc1e05a-c5d6-4b32-8a84-bab53f3c439f-kube-api-access-vm95t\") pod \"prometheus-operator-6f4946db56-rw6zb\" (UID: \"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:10.021093 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:10.021059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"
Sep 06 05:00:10.209343 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:10.209311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv" event={"ID":"c64b1fde-c4e4-49b4-a005-7e721932b165","Type":"ContainerStarted","Data":"5fe397041740f83d086e3fb994761845301b67e8d37adcfedd2011c1fc63a235"}
Sep 06 05:00:10.210522 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:10.210497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mx4tj" event={"ID":"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556","Type":"ContainerStarted","Data":"4a7990bfd78af96e58c3e3594212317fd36e90d0335ff434e1263abf28ee8b8f"}
Sep 06 05:00:10.211928 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:10.211907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f689b587d-grvvf" event={"ID":"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33","Type":"ContainerStarted","Data":"385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40"}
Sep 06 05:00:10.212036 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:10.211943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f689b587d-grvvf" event={"ID":"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33","Type":"ContainerStarted","Data":"bbb189692de10227c51b1afa597fd9ec353ada51ef92ee21c3fcd92f5c717401"}
Sep 06 05:00:10.212107 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:10.212053    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 05:00:10.224518 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:10.224478    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-dtkkv" podStartSLOduration=271.21499999 podStartE2EDuration="4m32.224464715s" totalImagesPullingTime="1.009464725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:10.223806316 +0000 UTC m=+286.365366269" watchObservedRunningTime="2026-09-06 05:00:10.224464715 +0000 UTC m=+286.366024656"
Sep 06 05:00:10.245605 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:10.245567    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f689b587d-grvvf" podStartSLOduration=286.245553871 podStartE2EDuration="4m46.245553871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:10.244732806 +0000 UTC m=+286.386292964" watchObservedRunningTime="2026-09-06 05:00:10.245553871 +0000 UTC m=+286.387113813"
Sep 06 05:00:10.301342 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:10.301273    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rw6zb"]
Sep 06 05:00:10.304918 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:10.304887    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc1e05a_c5d6_4b32_8a84_bab53f3c439f.slice/crio-ff8e7a49d84fcb4aa5520c6e4541736aa95b5d30ac956f68db2599c9821720c9 WatchSource:0}: Error finding container ff8e7a49d84fcb4aa5520c6e4541736aa95b5d30ac956f68db2599c9821720c9: Status 404 returned error can't find the container with id ff8e7a49d84fcb4aa5520c6e4541736aa95b5d30ac956f68db2599c9821720c9
Sep 06 05:00:11.218049 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:11.218010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb" event={"ID":"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f","Type":"ContainerStarted","Data":"ff8e7a49d84fcb4aa5520c6e4541736aa95b5d30ac956f68db2599c9821720c9"}
Sep 06 05:00:12.222876 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:12.222840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb" event={"ID":"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f","Type":"ContainerStarted","Data":"b76876210dc6c0df7f2fe7c0f8ea5af94b9040b37c6ec7b1a005ca0ca4c975f4"}
Sep 06 05:00:12.223318 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:12.222894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb" event={"ID":"6dc1e05a-c5d6-4b32-8a84-bab53f3c439f","Type":"ContainerStarted","Data":"6d34ffa52fdd969fc8072e9bc36d81f9a844d8495b401c033abbf0754c798867"}
Sep 06 05:00:12.224684 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:12.224661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mx4tj" event={"ID":"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556","Type":"ContainerStarted","Data":"01dff4c96988513683fd626b698804c123a3950cba28151f2efb57578746cb13"}
Sep 06 05:00:12.224814 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:12.224699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mx4tj" event={"ID":"84a76bc6-6f0f-4d5c-b48f-9e8ab6651556","Type":"ContainerStarted","Data":"7d4f8b107eae612dae19335e8043208d9c6c8aa8372cb343c6340dc0fe91254b"}
Sep 06 05:00:12.224814 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:12.224787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mx4tj"
Sep 06 05:00:12.239301 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:12.239258    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw6zb" podStartSLOduration=1.706897699 podStartE2EDuration="3.239244355s" totalImagesPullingTime="1.532346656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:12.238474963 +0000 UTC m=+288.380034915" watchObservedRunningTime="2026-09-06 05:00:12.239244355 +0000 UTC m=+288.380804298"
Sep 06 05:00:12.252621 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:12.252575    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mx4tj" podStartSLOduration=251.801062371 podStartE2EDuration="4m14.252561018s" totalImagesPullingTime="2.451498647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:12.251728348 +0000 UTC m=+288.393288305" watchObservedRunningTime="2026-09-06 05:00:12.252561018 +0000 UTC m=+288.394120959"
Sep 06 05:00:14.030495 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.030464    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"]
Sep 06 05:00:14.037734 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.037702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.037981 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.037950    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"]
Sep 06 05:00:14.040253 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.040226    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 06 05:00:14.040372 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.040264    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 06 05:00:14.040506 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.040485    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jkzqn\""
Sep 06 05:00:14.041894 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.041873    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8zj4c"]
Sep 06 05:00:14.042027 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.042009    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.044973 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.044954    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w4c9h\""
Sep 06 05:00:14.045386 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.045369    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 06 05:00:14.045821 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.045803    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 06 05:00:14.046128 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.046111    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 06 05:00:14.046854 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.046565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"]
Sep 06 05:00:14.046854 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.046608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"]
Sep 06 05:00:14.046854 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.046706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.048744 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.048691    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 06 05:00:14.049093 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.048961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 06 05:00:14.049235 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.049181    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 06 05:00:14.049437 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.049419    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6z6cl\""
Sep 06 05:00:14.138546 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzm9p\" (UniqueName: \"kubernetes.io/projected/bf388998-929b-40b8-99a0-0a9ed282507a-kube-api-access-lzm9p\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.138546 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v922z\" (UniqueName: \"kubernetes.io/projected/9fe134b3-19d7-4321-99ba-1addfd2cfae0-kube-api-access-v922z\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.138785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf388998-929b-40b8-99a0-0a9ed282507a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.138785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.138785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf388998-929b-40b8-99a0-0a9ed282507a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.138785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.138785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e91beee9-c6b0-4164-9751-5b85b4c08ab0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.138785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswrc\" (UniqueName: \"kubernetes.io/projected/e91beee9-c6b0-4164-9751-5b85b4c08ab0-kube-api-access-qswrc\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.138785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-accelerators-collector-config\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.138785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e91beee9-c6b0-4164-9751-5b85b4c08ab0-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.138785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fe134b3-19d7-4321-99ba-1addfd2cfae0-root\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.138785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-textfile\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.139281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e91beee9-c6b0-4164-9751-5b85b4c08ab0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.139281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-wtmp\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.139281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fe134b3-19d7-4321-99ba-1addfd2cfae0-sys\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.139281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.139281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-tls\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.139281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.139281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.138922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fe134b3-19d7-4321-99ba-1addfd2cfae0-metrics-client-ca\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.240053 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-tls\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.240270 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.240270 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fe134b3-19d7-4321-99ba-1addfd2cfae0-metrics-client-ca\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.240270 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzm9p\" (UniqueName: \"kubernetes.io/projected/bf388998-929b-40b8-99a0-0a9ed282507a-kube-api-access-lzm9p\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.240780 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v922z\" (UniqueName: \"kubernetes.io/projected/9fe134b3-19d7-4321-99ba-1addfd2cfae0-kube-api-access-v922z\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.240891 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf388998-929b-40b8-99a0-0a9ed282507a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.240891 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.241004 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.241004 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf388998-929b-40b8-99a0-0a9ed282507a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.241004 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.241004 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.240995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e91beee9-c6b0-4164-9751-5b85b4c08ab0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.241237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.241027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswrc\" (UniqueName: \"kubernetes.io/projected/e91beee9-c6b0-4164-9751-5b85b4c08ab0-kube-api-access-qswrc\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.241237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.241054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-accelerators-collector-config\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.241237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.241078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e91beee9-c6b0-4164-9751-5b85b4c08ab0-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.241237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.241112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fe134b3-19d7-4321-99ba-1addfd2cfae0-root\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.241237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.241141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-textfile\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.241471 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.241279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf388998-929b-40b8-99a0-0a9ed282507a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.241471 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:14.241341    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 06 05:00:14.241618 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.241598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fe134b3-19d7-4321-99ba-1addfd2cfae0-metrics-client-ca\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.241815 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.241794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf388998-929b-40b8-99a0-0a9ed282507a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.241936 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.241917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fe134b3-19d7-4321-99ba-1addfd2cfae0-root\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.242070 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.241927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e91beee9-c6b0-4164-9751-5b85b4c08ab0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.242179 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.242046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-textfile\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.242278 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:14.242108    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-tls podName:bf388998-929b-40b8-99a0-0a9ed282507a nodeName:}" failed. No retries permitted until 2026-09-06 05:00:14.742090207 +0000 UTC m=+290.883650136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-gsqlq" (UID: "bf388998-929b-40b8-99a0-0a9ed282507a") : secret "kube-state-metrics-tls" not found
Sep 06 05:00:14.242418 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.242405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-wtmp\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.242535 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.242521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fe134b3-19d7-4321-99ba-1addfd2cfae0-sys\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.242657 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.242642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.242864 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.242842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-wtmp\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.242939 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.242870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fe134b3-19d7-4321-99ba-1addfd2cfae0-sys\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.243117 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.243094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e91beee9-c6b0-4164-9751-5b85b4c08ab0-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.243669 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.243644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-accelerators-collector-config\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.243809 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.243760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-tls\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.243994 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:14.243980    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 06 05:00:14.244136 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:14.244125    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e91beee9-c6b0-4164-9751-5b85b4c08ab0-openshift-state-metrics-tls podName:e91beee9-c6b0-4164-9751-5b85b4c08ab0 nodeName:}" failed. No retries permitted until 2026-09-06 05:00:14.744107956 +0000 UTC m=+290.885667890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e91beee9-c6b0-4164-9751-5b85b4c08ab0-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-xgzf2" (UID: "e91beee9-c6b0-4164-9751-5b85b4c08ab0") : secret "openshift-state-metrics-tls" not found
Sep 06 05:00:14.244833 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.244811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e91beee9-c6b0-4164-9751-5b85b4c08ab0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.244961 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.244940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.245614 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.245595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fe134b3-19d7-4321-99ba-1addfd2cfae0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.247780 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.247751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v922z\" (UniqueName: \"kubernetes.io/projected/9fe134b3-19d7-4321-99ba-1addfd2cfae0-kube-api-access-v922z\") pod \"node-exporter-8zj4c\" (UID: \"9fe134b3-19d7-4321-99ba-1addfd2cfae0\") " pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.248026 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.248002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzm9p\" (UniqueName: \"kubernetes.io/projected/bf388998-929b-40b8-99a0-0a9ed282507a-kube-api-access-lzm9p\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.252073 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.252053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswrc\" (UniqueName: \"kubernetes.io/projected/e91beee9-c6b0-4164-9751-5b85b4c08ab0-kube-api-access-qswrc\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.365551 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.365471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8zj4c"
Sep 06 05:00:14.746603 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.746569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e91beee9-c6b0-4164-9751-5b85b4c08ab0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.746747 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.746651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.749104 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.749076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf388998-929b-40b8-99a0-0a9ed282507a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gsqlq\" (UID: \"bf388998-929b-40b8-99a0-0a9ed282507a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:14.749240 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.749135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e91beee9-c6b0-4164-9751-5b85b4c08ab0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xgzf2\" (UID: \"e91beee9-c6b0-4164-9751-5b85b4c08ab0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.952637 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.952606    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"
Sep 06 05:00:14.959527 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:14.959502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"
Sep 06 05:00:15.238011 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:15.237960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zj4c" event={"ID":"9fe134b3-19d7-4321-99ba-1addfd2cfae0","Type":"ContainerStarted","Data":"beb2224daa716c84d614cc66794408b5cb737517f4d7d976fae44f9132bdc200"}
Sep 06 05:00:15.241643 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:15.241616    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2"]
Sep 06 05:00:15.247097 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:15.247070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq"]
Sep 06 05:00:15.430140 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:15.430061    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf388998_929b_40b8_99a0_0a9ed282507a.slice/crio-0d1c0f52b786a7d29cf395054fe06c06d72bfbcc70135f232a5a1973eeccbf10 WatchSource:0}: Error finding container 0d1c0f52b786a7d29cf395054fe06c06d72bfbcc70135f232a5a1973eeccbf10: Status 404 returned error can't find the container with id 0d1c0f52b786a7d29cf395054fe06c06d72bfbcc70135f232a5a1973eeccbf10
Sep 06 05:00:15.430416 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:15.430391    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91beee9_c6b0_4164_9751_5b85b4c08ab0.slice/crio-d55e672fd577ade534959a33241b4e19788782413810144dc4e568a5714d9bec WatchSource:0}: Error finding container d55e672fd577ade534959a33241b4e19788782413810144dc4e568a5714d9bec: Status 404 returned error can't find the container with id d55e672fd577ade534959a33241b4e19788782413810144dc4e568a5714d9bec
Sep 06 05:00:16.245310 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:16.245083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2" event={"ID":"e91beee9-c6b0-4164-9751-5b85b4c08ab0","Type":"ContainerStarted","Data":"4ea6db1d6f6c9760fe058a18d806ee61abd6be842aad39d14a67a033e7fdeb1a"}
Sep 06 05:00:16.245310 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:16.245219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2" event={"ID":"e91beee9-c6b0-4164-9751-5b85b4c08ab0","Type":"ContainerStarted","Data":"6cfb6c37d1e9f4902ff6eef78a4df5f72d898801b4cac212ec70c1f00744cfbc"}
Sep 06 05:00:16.245310 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:16.245253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2" event={"ID":"e91beee9-c6b0-4164-9751-5b85b4c08ab0","Type":"ContainerStarted","Data":"d55e672fd577ade534959a33241b4e19788782413810144dc4e568a5714d9bec"}
Sep 06 05:00:16.247019 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:16.246990    2580 generic.go:356] "Generic (PLEG): container finished" podID="9fe134b3-19d7-4321-99ba-1addfd2cfae0" containerID="80f9a802632cea055599b833d426f340491aff9d1144c2e3fb6466fdf25a71f9" exitCode=0
Sep 06 05:00:16.247142 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:16.247072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zj4c" event={"ID":"9fe134b3-19d7-4321-99ba-1addfd2cfae0","Type":"ContainerDied","Data":"80f9a802632cea055599b833d426f340491aff9d1144c2e3fb6466fdf25a71f9"}
Sep 06 05:00:16.248714 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:16.248692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq" event={"ID":"bf388998-929b-40b8-99a0-0a9ed282507a","Type":"ContainerStarted","Data":"0d1c0f52b786a7d29cf395054fe06c06d72bfbcc70135f232a5a1973eeccbf10"}
Sep 06 05:00:17.255526 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:17.255493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2" event={"ID":"e91beee9-c6b0-4164-9751-5b85b4c08ab0","Type":"ContainerStarted","Data":"1814fb5d35867c9a1a31997b9eebcdc596cbf60f65e96edb603f63db7c6e856b"}
Sep 06 05:00:17.257713 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:17.257685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zj4c" event={"ID":"9fe134b3-19d7-4321-99ba-1addfd2cfae0","Type":"ContainerStarted","Data":"1009f37f95f9d21e798dc97ecc036ad941abe9a5635f1e65106ded61369324ad"}
Sep 06 05:00:17.257836 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:17.257735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zj4c" event={"ID":"9fe134b3-19d7-4321-99ba-1addfd2cfae0","Type":"ContainerStarted","Data":"2a54a86dc91dbdd1e1eff73e8c82cab1fd47f92be258e2d35dc14018a7a9b6be"}
Sep 06 05:00:17.259985 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:17.259964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq" event={"ID":"bf388998-929b-40b8-99a0-0a9ed282507a","Type":"ContainerStarted","Data":"084ca37cc5784093333bc6a0528b4fff006b1d70b9b0ab0cf6d9d957c3d2f3ca"}
Sep 06 05:00:17.260116 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:17.260014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq" event={"ID":"bf388998-929b-40b8-99a0-0a9ed282507a","Type":"ContainerStarted","Data":"53d451f79b86af030585c6633fdcdb4dee73ecb8b37a55061d243a2f76b15c83"}
Sep 06 05:00:17.260116 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:17.260041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq" event={"ID":"bf388998-929b-40b8-99a0-0a9ed282507a","Type":"ContainerStarted","Data":"fda6dc2de8cc99e35fc50738b8343e4fccc8f5c956ccfeb7736d36507eec4236"}
Sep 06 05:00:17.291596 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:17.291540    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gsqlq" podStartSLOduration=2.003991308 podStartE2EDuration="3.291521006s" totalImagesPullingTime="1.287529698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:17.291391678 +0000 UTC m=+293.432951642" watchObservedRunningTime="2026-09-06 05:00:17.291521006 +0000 UTC m=+293.433080951"
Sep 06 05:00:17.292499 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:17.292467    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xgzf2" podStartSLOduration=2.124397221 podStartE2EDuration="3.29245592s" totalImagesPullingTime="1.168058699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:17.27059022 +0000 UTC m=+293.412150163" watchObservedRunningTime="2026-09-06 05:00:17.29245592 +0000 UTC m=+293.434015862"
Sep 06 05:00:17.306747 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:17.306682    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8zj4c" podStartSLOduration=2.14130993 podStartE2EDuration="3.306667412s" totalImagesPullingTime="1.093642699s" totalInitContainerRuntime="71.714783ms" isStatefulPod=true podCreationTimestamp="2026-09-06 05:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:17.306383476 +0000 UTC m=+293.447943449" watchObservedRunningTime="2026-09-06 05:00:17.306667412 +0000 UTC m=+293.448227356"
Sep 06 05:00:18.342373 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:18.342331    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-csrcb" podUID="d3a5a3f7-dc34-4f63-9fa9-5660515f8000"
Sep 06 05:00:18.348787 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.348761    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b64f578b7-8zvff"]
Sep 06 05:00:18.351964 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.351949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.354404 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.354381    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-le8uscvh0bdq\""
Sep 06 05:00:18.354404 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.354399    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 06 05:00:18.355228 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.355207    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 06 05:00:18.355228 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.355223    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 06 05:00:18.355452 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.355430    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 06 05:00:18.355572 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.355474    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ckkhq\""
Sep 06 05:00:18.361404 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.361382    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b64f578b7-8zvff"]
Sep 06 05:00:18.481880 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.481835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c25ce7ff-6a88-4eba-8c76-81096a80ca14-metrics-server-audit-profiles\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.481880 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.481885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c25ce7ff-6a88-4eba-8c76-81096a80ca14-secret-metrics-server-client-certs\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.482085 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.481900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c25ce7ff-6a88-4eba-8c76-81096a80ca14-client-ca-bundle\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.482085 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.481938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c25ce7ff-6a88-4eba-8c76-81096a80ca14-secret-metrics-server-tls\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.482085 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.481977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c8kh\" (UniqueName: \"kubernetes.io/projected/c25ce7ff-6a88-4eba-8c76-81096a80ca14-kube-api-access-5c8kh\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.482085 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.482033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25ce7ff-6a88-4eba-8c76-81096a80ca14-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.482085 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.482071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c25ce7ff-6a88-4eba-8c76-81096a80ca14-audit-log\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.583002 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.582963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8kh\" (UniqueName: \"kubernetes.io/projected/c25ce7ff-6a88-4eba-8c76-81096a80ca14-kube-api-access-5c8kh\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.583177 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.583017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25ce7ff-6a88-4eba-8c76-81096a80ca14-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.583177 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.583042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c25ce7ff-6a88-4eba-8c76-81096a80ca14-audit-log\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.583177 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.583067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c25ce7ff-6a88-4eba-8c76-81096a80ca14-metrics-server-audit-profiles\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.583177 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.583114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c25ce7ff-6a88-4eba-8c76-81096a80ca14-secret-metrics-server-client-certs\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.583177 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.583134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c25ce7ff-6a88-4eba-8c76-81096a80ca14-client-ca-bundle\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.583427 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.583217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c25ce7ff-6a88-4eba-8c76-81096a80ca14-secret-metrics-server-tls\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.583520 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.583500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c25ce7ff-6a88-4eba-8c76-81096a80ca14-audit-log\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.583790 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.583762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25ce7ff-6a88-4eba-8c76-81096a80ca14-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.584395 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.584367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c25ce7ff-6a88-4eba-8c76-81096a80ca14-metrics-server-audit-profiles\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.585722 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.585693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c25ce7ff-6a88-4eba-8c76-81096a80ca14-secret-metrics-server-client-certs\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.585816 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.585750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c25ce7ff-6a88-4eba-8c76-81096a80ca14-secret-metrics-server-tls\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.585858 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.585823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c25ce7ff-6a88-4eba-8c76-81096a80ca14-client-ca-bundle\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.590140 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.590120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c8kh\" (UniqueName: \"kubernetes.io/projected/c25ce7ff-6a88-4eba-8c76-81096a80ca14-kube-api-access-5c8kh\") pod \"metrics-server-5b64f578b7-8zvff\" (UID: \"c25ce7ff-6a88-4eba-8c76-81096a80ca14\") " pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.661823 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.661741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:18.807564 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.805080    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"]
Sep 06 05:00:18.810552 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.810528    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:18.812827 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.812805    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-b2fcl\""
Sep 06 05:00:18.813019 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.812997    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 06 05:00:18.813482 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.813461    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"]
Sep 06 05:00:18.886336 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.886305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvx86\" (UniqueName: \"kubernetes.io/projected/8bbef036-9031-4ad0-a729-438c9ff26410-kube-api-access-rvx86\") pod \"monitoring-plugin-85bbbbbf4-wnlzt\" (UID: \"8bbef036-9031-4ad0-a729-438c9ff26410\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:18.886336 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.886339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bbef036-9031-4ad0-a729-438c9ff26410-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-wnlzt\" (UID: \"8bbef036-9031-4ad0-a729-438c9ff26410\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:18.932337 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.932308    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b64f578b7-8zvff"]
Sep 06 05:00:18.934591 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:18.934563    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25ce7ff_6a88_4eba_8c76_81096a80ca14.slice/crio-29ba556bb5c1bc84e3de14ae92dc5254e6ca3572e6db2b0afdf708e39c1c1d1b WatchSource:0}: Error finding container 29ba556bb5c1bc84e3de14ae92dc5254e6ca3572e6db2b0afdf708e39c1c1d1b: Status 404 returned error can't find the container with id 29ba556bb5c1bc84e3de14ae92dc5254e6ca3572e6db2b0afdf708e39c1c1d1b
Sep 06 05:00:18.986784 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.986744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvx86\" (UniqueName: \"kubernetes.io/projected/8bbef036-9031-4ad0-a729-438c9ff26410-kube-api-access-rvx86\") pod \"monitoring-plugin-85bbbbbf4-wnlzt\" (UID: \"8bbef036-9031-4ad0-a729-438c9ff26410\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:18.986908 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.986793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bbef036-9031-4ad0-a729-438c9ff26410-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-wnlzt\" (UID: \"8bbef036-9031-4ad0-a729-438c9ff26410\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:18.986981 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:18.986923    2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 06 05:00:18.987039 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:18.986982    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bbef036-9031-4ad0-a729-438c9ff26410-monitoring-plugin-cert podName:8bbef036-9031-4ad0-a729-438c9ff26410 nodeName:}" failed. No retries permitted until 2026-09-06 05:00:19.486961768 +0000 UTC m=+295.628521702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/8bbef036-9031-4ad0-a729-438c9ff26410-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-wnlzt" (UID: "8bbef036-9031-4ad0-a729-438c9ff26410") : secret "monitoring-plugin-cert" not found
Sep 06 05:00:18.994790 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:18.994765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvx86\" (UniqueName: \"kubernetes.io/projected/8bbef036-9031-4ad0-a729-438c9ff26410-kube-api-access-rvx86\") pod \"monitoring-plugin-85bbbbbf4-wnlzt\" (UID: \"8bbef036-9031-4ad0-a729-438c9ff26410\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:19.211248 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.211218    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"]
Sep 06 05:00:19.214893 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.214876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.217118 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.217095    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 06 05:00:19.217273 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.217095    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 06 05:00:19.217273 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.217143    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kxtf6\""
Sep 06 05:00:19.217273 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.217222    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 06 05:00:19.217273 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.217221    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 06 05:00:19.217424 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.217144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 06 05:00:19.221776 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.221757    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 06 05:00:19.225086 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.225063    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"]
Sep 06 05:00:19.267993 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.267962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff" event={"ID":"c25ce7ff-6a88-4eba-8c76-81096a80ca14","Type":"ContainerStarted","Data":"29ba556bb5c1bc84e3de14ae92dc5254e6ca3572e6db2b0afdf708e39c1c1d1b"}
Sep 06 05:00:19.289994 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.289968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54a756ad-0150-4934-a646-14f5ed91d82e-metrics-client-ca\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.290120 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.290008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-federate-client-tls\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.290120 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.290024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-telemeter-client-tls\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.290120 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.290041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.290258 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.290146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-secret-telemeter-client\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.290258 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.290201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a756ad-0150-4934-a646-14f5ed91d82e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.290258 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.290232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrxn4\" (UniqueName: \"kubernetes.io/projected/54a756ad-0150-4934-a646-14f5ed91d82e-kube-api-access-xrxn4\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.290349 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.290300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a756ad-0150-4934-a646-14f5ed91d82e-serving-certs-ca-bundle\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.391426 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.391395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrxn4\" (UniqueName: \"kubernetes.io/projected/54a756ad-0150-4934-a646-14f5ed91d82e-kube-api-access-xrxn4\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.391861 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.391454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a756ad-0150-4934-a646-14f5ed91d82e-serving-certs-ca-bundle\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.391861 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.391482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54a756ad-0150-4934-a646-14f5ed91d82e-metrics-client-ca\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.391861 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.391510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-federate-client-tls\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.391861 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.391528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-telemeter-client-tls\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.391861 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.391546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.391861 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.391587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-secret-telemeter-client\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.391861 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.391610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a756ad-0150-4934-a646-14f5ed91d82e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.392483 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.392415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a756ad-0150-4934-a646-14f5ed91d82e-serving-certs-ca-bundle\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.392483 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.392459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a756ad-0150-4934-a646-14f5ed91d82e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.393025 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.392998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54a756ad-0150-4934-a646-14f5ed91d82e-metrics-client-ca\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.394289 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.394271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.394451 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.394435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-telemeter-client-tls\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.394626 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.394608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-secret-telemeter-client\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.394748 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.394730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/54a756ad-0150-4934-a646-14f5ed91d82e-federate-client-tls\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.398617 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.398599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrxn4\" (UniqueName: \"kubernetes.io/projected/54a756ad-0150-4934-a646-14f5ed91d82e-kube-api-access-xrxn4\") pod \"telemeter-client-6ffd9c6b55-4s8q7\" (UID: \"54a756ad-0150-4934-a646-14f5ed91d82e\") " pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.493092 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.493011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bbef036-9031-4ad0-a729-438c9ff26410-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-wnlzt\" (UID: \"8bbef036-9031-4ad0-a729-438c9ff26410\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:19.495896 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.495872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bbef036-9031-4ad0-a729-438c9ff26410-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-wnlzt\" (UID: \"8bbef036-9031-4ad0-a729-438c9ff26410\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:19.524724 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.524693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"
Sep 06 05:00:19.724326 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.724289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:19.801599 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:19.801569    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7"]
Sep 06 05:00:19.805503 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:19.805477    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a756ad_0150_4934_a646_14f5ed91d82e.slice/crio-9766f60fdc9b6406e7b2e3edc8ba79abb5dc54a695b9b42c37ddfc7d140c2982 WatchSource:0}: Error finding container 9766f60fdc9b6406e7b2e3edc8ba79abb5dc54a695b9b42c37ddfc7d140c2982: Status 404 returned error can't find the container with id 9766f60fdc9b6406e7b2e3edc8ba79abb5dc54a695b9b42c37ddfc7d140c2982
Sep 06 05:00:20.004496 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:20.004418    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"]
Sep 06 05:00:20.276207 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:20.275963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7" event={"ID":"54a756ad-0150-4934-a646-14f5ed91d82e","Type":"ContainerStarted","Data":"9766f60fdc9b6406e7b2e3edc8ba79abb5dc54a695b9b42c37ddfc7d140c2982"}
Sep 06 05:00:20.382743 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:20.382703    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbef036_9031_4ad0_a729_438c9ff26410.slice/crio-b35605e322823f4b10f7b0fb7198c5eede4dbc75ebceb4565f63fc961dd6a9a0 WatchSource:0}: Error finding container b35605e322823f4b10f7b0fb7198c5eede4dbc75ebceb4565f63fc961dd6a9a0: Status 404 returned error can't find the container with id b35605e322823f4b10f7b0fb7198c5eede4dbc75ebceb4565f63fc961dd6a9a0
Sep 06 05:00:21.281381 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:21.281350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt" event={"ID":"8bbef036-9031-4ad0-a729-438c9ff26410","Type":"ContainerStarted","Data":"b35605e322823f4b10f7b0fb7198c5eede4dbc75ebceb4565f63fc961dd6a9a0"}
Sep 06 05:00:21.283056 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:21.283029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff" event={"ID":"c25ce7ff-6a88-4eba-8c76-81096a80ca14","Type":"ContainerStarted","Data":"6611eca15c3642d442bacee75082604981ab0875f3b564186d4482f9cc30a956"}
Sep 06 05:00:21.300304 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:21.300257    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff" podStartSLOduration=1.802505224 podStartE2EDuration="3.300241631s" totalImagesPullingTime="1.497736407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:21.298632176 +0000 UTC m=+297.440192145" watchObservedRunningTime="2026-09-06 05:00:21.300241631 +0000 UTC m=+297.441801573"
Sep 06 05:00:22.229573 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:22.229545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mx4tj"
Sep 06 05:00:22.289100 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:22.289068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7" event={"ID":"54a756ad-0150-4934-a646-14f5ed91d82e","Type":"ContainerStarted","Data":"68af96a56a2c21ed9d8d92801d06271b929fb980f880e908f3509a2201659185"}
Sep 06 05:00:22.290610 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:22.290588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt" event={"ID":"8bbef036-9031-4ad0-a729-438c9ff26410","Type":"ContainerStarted","Data":"7709234b01d0affc5d30adae57a3597cec75f7afde5d2dc9804ceaab776a60b8"}
Sep 06 05:00:22.290915 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:22.290894    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:22.295578 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:22.295551    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt"
Sep 06 05:00:22.305713 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:22.305669    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-wnlzt" podStartSLOduration=2.63069868 podStartE2EDuration="4.305654942s" totalImagesPullingTime="1.674956262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:22.303949918 +0000 UTC m=+298.445509892" watchObservedRunningTime="2026-09-06 05:00:22.305654942 +0000 UTC m=+298.447214883"
Sep 06 05:00:23.296918 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:23.296826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7" event={"ID":"54a756ad-0150-4934-a646-14f5ed91d82e","Type":"ContainerStarted","Data":"d7efe8bd2f62d50302a8e2fc7f09984bb197af0dfb4db6b03f32a6f2d1426545"}
Sep 06 05:00:23.296918 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:23.296883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7" event={"ID":"54a756ad-0150-4934-a646-14f5ed91d82e","Type":"ContainerStarted","Data":"4ce4525332fc6bc9554f3dcb4af5512327715e4e505d88fb497b006a1ac4148b"}
Sep 06 05:00:23.317344 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:23.317302    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6ffd9c6b55-4s8q7" podStartSLOduration=1.186916102 podStartE2EDuration="4.31728809s" totalImagesPullingTime="3.130371988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:00:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:23.314917417 +0000 UTC m=+299.456477367" watchObservedRunningTime="2026-09-06 05:00:23.31728809 +0000 UTC m=+299.458848032"
Sep 06 05:00:24.313305 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:24.313276    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:00:24.317758 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:24.317735    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:00:24.319787 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:24.319768    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 06 05:00:29.106639 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:29.106605    2580 patch_prober.go:28] interesting pod/image-registry-6f689b587d-grvvf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 06 05:00:29.108843 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:29.106657    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f689b587d-grvvf" podUID="531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 05:00:29.341737 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:29.341707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 05:00:29.376128 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:29.376063    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f689b587d-grvvf"]
Sep 06 05:00:29.380283 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:29.380251    2580 patch_prober.go:28] interesting pod/image-registry-6f689b587d-grvvf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 06 05:00:29.380464 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:29.380312    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6f689b587d-grvvf" podUID="531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 05:00:38.662459 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:38.662424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:38.662846 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:38.662502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:39.380202 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:39.380149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 05:00:54.218104 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.218066    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bc95747bc-kfzpx"]
Sep 06 05:00:54.222358 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.222338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.224599 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.224573    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 06 05:00:54.224702 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.224654    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b6968\""
Sep 06 05:00:54.224877 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.224864    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 06 05:00:54.225467 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.225431    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 06 05:00:54.225569 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.225450    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 06 05:00:54.225640 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.225460    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 06 05:00:54.225640 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.225475    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 06 05:00:54.225988 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.225960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 06 05:00:54.229052 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.229029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 06 05:00:54.232130 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.232109    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc95747bc-kfzpx"]
Sep 06 05:00:54.288764 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.288741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-oauth-serving-cert\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.288892 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.288771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-serving-cert\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.288892 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.288805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-console-config\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.288892 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.288875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77xvl\" (UniqueName: \"kubernetes.io/projected/7aec1bdb-9206-4093-94dc-a024b4db4239-kube-api-access-77xvl\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.289014 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.288915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-service-ca\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.289014 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.288938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-trusted-ca-bundle\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.289014 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.288958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-oauth-config\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.389399 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.389376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-oauth-serving-cert\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.389399 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.389406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-serving-cert\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.389623 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.389571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-console-config\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.389623 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.389618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77xvl\" (UniqueName: \"kubernetes.io/projected/7aec1bdb-9206-4093-94dc-a024b4db4239-kube-api-access-77xvl\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.389702 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.389666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-service-ca\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.389702 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.389691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-trusted-ca-bundle\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.389763 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.389714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-oauth-config\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.390133 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.390108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-oauth-serving-cert\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.390600 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.390562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-service-ca\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.390766 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.390742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-console-config\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.391029 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.391011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-trusted-ca-bundle\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.392359 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.392339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-serving-cert\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.392588 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.392570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-oauth-config\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.396283 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.396234    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f689b587d-grvvf" podUID="531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" containerName="registry" containerID="cri-o://385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40" gracePeriod=30
Sep 06 05:00:54.397753 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.397728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77xvl\" (UniqueName: \"kubernetes.io/projected/7aec1bdb-9206-4093-94dc-a024b4db4239-kube-api-access-77xvl\") pod \"console-6bc95747bc-kfzpx\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") " pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.532693 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.532656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:00:54.638102 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.638078    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 05:00:54.692074 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.692042    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") pod \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") "
Sep 06 05:00:54.692243 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.692094    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-bound-sa-token\") pod \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") "
Sep 06 05:00:54.692243 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.692132    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-image-registry-private-configuration\") pod \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") "
Sep 06 05:00:54.692243 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.692176    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-ca-trust-extracted\") pod \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") "
Sep 06 05:00:54.692341 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.692324    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-certificates\") pod \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") "
Sep 06 05:00:54.692394 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.692368    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-installation-pull-secrets\") pod \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") "
Sep 06 05:00:54.692448 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.692402    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjzh4\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-kube-api-access-zjzh4\") pod \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") "
Sep 06 05:00:54.692448 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.692435    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-trusted-ca\") pod \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\" (UID: \"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33\") "
Sep 06 05:00:54.692768 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.692704    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:00:54.693191 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.693074    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:00:54.694897 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.694869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-kube-api-access-zjzh4" (OuterVolumeSpecName: "kube-api-access-zjzh4") pod "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33"). InnerVolumeSpecName "kube-api-access-zjzh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:00:54.694994 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.694918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:00:54.694994 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.694969    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:00:54.695077 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.695033    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:00:54.695077 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.695063    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:00:54.700787 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.700766    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" (UID: "531caa8b-f2b0-41f1-9c91-b68f8fbbfa33"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:00:54.793891 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.793867    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-installation-pull-secrets\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:00:54.793891 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.793891    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjzh4\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-kube-api-access-zjzh4\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:00:54.794015 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.793900    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-trusted-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:00:54.794015 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.793909    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:00:54.794015 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.793917    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-bound-sa-token\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:00:54.794015 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.793925    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-image-registry-private-configuration\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:00:54.794015 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.793934    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-ca-trust-extracted\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:00:54.794015 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.793942    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33-registry-certificates\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:00:54.805399 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.805380    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc95747bc-kfzpx"]
Sep 06 05:00:54.838074 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:00:54.838050    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aec1bdb_9206_4093_94dc_a024b4db4239.slice/crio-c7008de9ade67d912f2309ec1639755f7420ed8dfb639924c898e69b65f0089b WatchSource:0}: Error finding container c7008de9ade67d912f2309ec1639755f7420ed8dfb639924c898e69b65f0089b: Status 404 returned error can't find the container with id c7008de9ade67d912f2309ec1639755f7420ed8dfb639924c898e69b65f0089b
Sep 06 05:00:54.839950 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:54.839936    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 05:00:55.422023 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.421987    2580 generic.go:356] "Generic (PLEG): container finished" podID="a21526c5-2c0d-48dc-89c1-11b2abd6dac3" containerID="687e0eb5a0852d41ec84b66c7bfc760c651df87eea2306d17f5e3c46eb90c7cc" exitCode=0
Sep 06 05:00:55.422471 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.422085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5" event={"ID":"a21526c5-2c0d-48dc-89c1-11b2abd6dac3","Type":"ContainerDied","Data":"687e0eb5a0852d41ec84b66c7bfc760c651df87eea2306d17f5e3c46eb90c7cc"}
Sep 06 05:00:55.422531 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.422496    2580 scope.go:117] "RemoveContainer" containerID="687e0eb5a0852d41ec84b66c7bfc760c651df87eea2306d17f5e3c46eb90c7cc"
Sep 06 05:00:55.423970 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.423941    2580 generic.go:356] "Generic (PLEG): container finished" podID="531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" containerID="385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40" exitCode=0
Sep 06 05:00:55.424102 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.423975    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f689b587d-grvvf"
Sep 06 05:00:55.424102 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.424044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f689b587d-grvvf" event={"ID":"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33","Type":"ContainerDied","Data":"385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40"}
Sep 06 05:00:55.424102 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.424080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f689b587d-grvvf" event={"ID":"531caa8b-f2b0-41f1-9c91-b68f8fbbfa33","Type":"ContainerDied","Data":"bbb189692de10227c51b1afa597fd9ec353ada51ef92ee21c3fcd92f5c717401"}
Sep 06 05:00:55.424295 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.424106    2580 scope.go:117] "RemoveContainer" containerID="385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40"
Sep 06 05:00:55.425717 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.425695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc95747bc-kfzpx" event={"ID":"7aec1bdb-9206-4093-94dc-a024b4db4239","Type":"ContainerStarted","Data":"c7008de9ade67d912f2309ec1639755f7420ed8dfb639924c898e69b65f0089b"}
Sep 06 05:00:55.434942 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.434922    2580 scope.go:117] "RemoveContainer" containerID="385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40"
Sep 06 05:00:55.435385 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:00:55.435350    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40\": container with ID starting with 385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40 not found: ID does not exist" containerID="385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40"
Sep 06 05:00:55.435486 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.435394    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40"} err="failed to get container status \"385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40\": rpc error: code = NotFound desc = could not find container \"385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40\": container with ID starting with 385d8099f33de235f99c37da72cc0d3c4c490781903b4b38f4dc70d8c5946f40 not found: ID does not exist"
Sep 06 05:00:55.450233 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.450209    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f689b587d-grvvf"]
Sep 06 05:00:55.457320 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:55.457295    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f689b587d-grvvf"]
Sep 06 05:00:56.347493 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:56.347462    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" path="/var/lib/kubelet/pods/531caa8b-f2b0-41f1-9c91-b68f8fbbfa33/volumes"
Sep 06 05:00:56.432828 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:56.432795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-x6wn5" event={"ID":"a21526c5-2c0d-48dc-89c1-11b2abd6dac3","Type":"ContainerStarted","Data":"e2261aa87c765829683687f2561bf6d85ce35d01a8fde30cef94d3115b7112ce"}
Sep 06 05:00:58.443516 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:58.443475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc95747bc-kfzpx" event={"ID":"7aec1bdb-9206-4093-94dc-a024b4db4239","Type":"ContainerStarted","Data":"e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950"}
Sep 06 05:00:58.461184 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:58.460975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bc95747bc-kfzpx" podStartSLOduration=1.719526166 podStartE2EDuration="4.460955995s" totalImagesPullingTime="2.741429829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:00:58.458743043 +0000 UTC m=+334.600303019" watchObservedRunningTime="2026-09-06 05:00:58.460955995 +0000 UTC m=+334.602515938"
Sep 06 05:00:58.667814 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:58.667786    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:00:58.671535 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:00:58.671513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b64f578b7-8zvff"
Sep 06 05:01:02.788854 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:02.788815    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gsqlq_bf388998-929b-40b8-99a0-0a9ed282507a/kube-state-metrics/0.log"
Sep 06 05:01:02.987042 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:02.987014    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gsqlq_bf388998-929b-40b8-99a0-0a9ed282507a/kube-rbac-proxy-main/0.log"
Sep 06 05:01:03.187055 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:03.186983    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gsqlq_bf388998-929b-40b8-99a0-0a9ed282507a/kube-rbac-proxy-self/0.log"
Sep 06 05:01:03.387598 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:03.387567    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b64f578b7-8zvff_c25ce7ff-6a88-4eba-8c76-81096a80ca14/metrics-server/0.log"
Sep 06 05:01:03.587889 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:03.587851    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-wnlzt_8bbef036-9031-4ad0-a729-438c9ff26410/monitoring-plugin/0.log"
Sep 06 05:01:03.787134 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:03.787107    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zj4c_9fe134b3-19d7-4321-99ba-1addfd2cfae0/init-textfile/0.log"
Sep 06 05:01:03.988974 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:03.988942    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zj4c_9fe134b3-19d7-4321-99ba-1addfd2cfae0/node-exporter/0.log"
Sep 06 05:01:04.188061 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:04.188035    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zj4c_9fe134b3-19d7-4321-99ba-1addfd2cfae0/kube-rbac-proxy/0.log"
Sep 06 05:01:04.533550 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:04.533517    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:01:04.533711 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:04.533570    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:01:04.538565 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:04.538543    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:01:05.475690 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:05.475661    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:01:05.588584 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:05.588557    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xgzf2_e91beee9-c6b0-4164-9751-5b85b4c08ab0/kube-rbac-proxy-main/0.log"
Sep 06 05:01:05.786561 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:05.786490    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xgzf2_e91beee9-c6b0-4164-9751-5b85b4c08ab0/kube-rbac-proxy-self/0.log"
Sep 06 05:01:05.986937 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:05.986902    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xgzf2_e91beee9-c6b0-4164-9751-5b85b4c08ab0/openshift-state-metrics/0.log"
Sep 06 05:01:07.589594 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:07.589560    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-rw6zb_6dc1e05a-c5d6-4b32-8a84-bab53f3c439f/prometheus-operator/0.log"
Sep 06 05:01:07.787441 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:07.787405    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-rw6zb_6dc1e05a-c5d6-4b32-8a84-bab53f3c439f/kube-rbac-proxy/0.log"
Sep 06 05:01:08.187771 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:08.187735    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ffd9c6b55-4s8q7_54a756ad-0150-4934-a646-14f5ed91d82e/telemeter-client/0.log"
Sep 06 05:01:08.387431 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:08.387397    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ffd9c6b55-4s8q7_54a756ad-0150-4934-a646-14f5ed91d82e/reload/0.log"
Sep 06 05:01:08.587367 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:08.587340    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ffd9c6b55-4s8q7_54a756ad-0150-4934-a646-14f5ed91d82e/kube-rbac-proxy/0.log"
Sep 06 05:01:09.986487 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:09.986448    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-dtkkv_c64b1fde-c4e4-49b4-a005-7e721932b165/networking-console-plugin/0.log"
Sep 06 05:01:10.491343 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:10.491313    2580 generic.go:356] "Generic (PLEG): container finished" podID="c1f7670f-7713-48fc-acaa-81b99459a98b" containerID="22aa7ab0063008b709d3421fb534221037de9683791ef5dd094fffbbb60f6dd4" exitCode=0
Sep 06 05:01:10.491507 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:10.491363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs" event={"ID":"c1f7670f-7713-48fc-acaa-81b99459a98b","Type":"ContainerDied","Data":"22aa7ab0063008b709d3421fb534221037de9683791ef5dd094fffbbb60f6dd4"}
Sep 06 05:01:10.491626 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:10.491614    2580 scope.go:117] "RemoveContainer" containerID="22aa7ab0063008b709d3421fb534221037de9683791ef5dd094fffbbb60f6dd4"
Sep 06 05:01:10.788057 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:10.787973    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bc95747bc-kfzpx_7aec1bdb-9206-4093-94dc-a024b4db4239/console/0.log"
Sep 06 05:01:11.496710 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:11.496673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-pnlgs" event={"ID":"c1f7670f-7713-48fc-acaa-81b99459a98b","Type":"ContainerStarted","Data":"4363d16ee9f48598660629875ccedb804bcb9ef431a08765083efe4a747c9a63"}
Sep 06 05:01:38.239386 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:38.239336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 05:01:38.241731 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:38.241712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a5a3f7-dc34-4f63-9fa9-5660515f8000-metrics-certs\") pod \"network-metrics-daemon-csrcb\" (UID: \"d3a5a3f7-dc34-4f63-9fa9-5660515f8000\") " pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 05:01:38.344327 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:38.344292    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b9kvk\""
Sep 06 05:01:38.352872 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:38.352851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csrcb"
Sep 06 05:01:38.631363 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:38.631290    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-csrcb"]
Sep 06 05:01:39.617648 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:39.617608    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-csrcb" event={"ID":"d3a5a3f7-dc34-4f63-9fa9-5660515f8000","Type":"ContainerStarted","Data":"2d1c1c44412e7324f40b4f3e93faf637a98f07b48e648bfb1d630a06713f401b"}
Sep 06 05:01:40.623212 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:40.623179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-csrcb" event={"ID":"d3a5a3f7-dc34-4f63-9fa9-5660515f8000","Type":"ContainerStarted","Data":"b5fdf53aa275cb9c89a9c4e665d9225eecee715712ea50a7d5365b3d0cdc862a"}
Sep 06 05:01:40.623580 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:40.623241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-csrcb" event={"ID":"d3a5a3f7-dc34-4f63-9fa9-5660515f8000","Type":"ContainerStarted","Data":"fbcddfaf4f768c75c011ddbdb810d27cca80e396021fc7063ae2ca89c52f0994"}
Sep 06 05:01:40.637918 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:40.637871    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-csrcb" podStartSLOduration=375.585434966 podStartE2EDuration="6m16.637857054s" totalImagesPullingTime="1.052422088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 04:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:01:40.636247411 +0000 UTC m=+376.777807364" watchObservedRunningTime="2026-09-06 05:01:40.637857054 +0000 UTC m=+376.779416997"
Sep 06 05:01:43.696041 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:01:43.696007    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bc95747bc-kfzpx"]
Sep 06 05:02:08.716489 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:08.716434    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bc95747bc-kfzpx" podUID="7aec1bdb-9206-4093-94dc-a024b4db4239" containerName="console" containerID="cri-o://e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950" gracePeriod=15
Sep 06 05:02:08.964877 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:08.964855    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:02:09.055593 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.055531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77xvl\" (UniqueName: \"kubernetes.io/projected/7aec1bdb-9206-4093-94dc-a024b4db4239-kube-api-access-77xvl\") pod \"7aec1bdb-9206-4093-94dc-a024b4db4239\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") "
Sep 06 05:02:09.055593 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.055565    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-service-ca\") pod \"7aec1bdb-9206-4093-94dc-a024b4db4239\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") "
Sep 06 05:02:09.055593 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.055586    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-trusted-ca-bundle\") pod \"7aec1bdb-9206-4093-94dc-a024b4db4239\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") "
Sep 06 05:02:09.055834 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.055607    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-console-config\") pod \"7aec1bdb-9206-4093-94dc-a024b4db4239\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") "
Sep 06 05:02:09.055834 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.055623    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-oauth-config\") pod \"7aec1bdb-9206-4093-94dc-a024b4db4239\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") "
Sep 06 05:02:09.055834 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.055659    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-oauth-serving-cert\") pod \"7aec1bdb-9206-4093-94dc-a024b4db4239\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") "
Sep 06 05:02:09.055834 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.055727    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-serving-cert\") pod \"7aec1bdb-9206-4093-94dc-a024b4db4239\" (UID: \"7aec1bdb-9206-4093-94dc-a024b4db4239\") "
Sep 06 05:02:09.056029 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.055999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-console-config" (OuterVolumeSpecName: "console-config") pod "7aec1bdb-9206-4093-94dc-a024b4db4239" (UID: "7aec1bdb-9206-4093-94dc-a024b4db4239"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:02:09.056029 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.055993    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-service-ca" (OuterVolumeSpecName: "service-ca") pod "7aec1bdb-9206-4093-94dc-a024b4db4239" (UID: "7aec1bdb-9206-4093-94dc-a024b4db4239"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:02:09.056117 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.056030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7aec1bdb-9206-4093-94dc-a024b4db4239" (UID: "7aec1bdb-9206-4093-94dc-a024b4db4239"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:02:09.056375 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.056355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7aec1bdb-9206-4093-94dc-a024b4db4239" (UID: "7aec1bdb-9206-4093-94dc-a024b4db4239"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:02:09.057976 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.057955    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7aec1bdb-9206-4093-94dc-a024b4db4239" (UID: "7aec1bdb-9206-4093-94dc-a024b4db4239"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:02:09.058060 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.058004    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aec1bdb-9206-4093-94dc-a024b4db4239-kube-api-access-77xvl" (OuterVolumeSpecName: "kube-api-access-77xvl") pod "7aec1bdb-9206-4093-94dc-a024b4db4239" (UID: "7aec1bdb-9206-4093-94dc-a024b4db4239"). InnerVolumeSpecName "kube-api-access-77xvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:02:09.058060 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.058049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7aec1bdb-9206-4093-94dc-a024b4db4239" (UID: "7aec1bdb-9206-4093-94dc-a024b4db4239"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:02:09.156818 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.156795    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:09.156818 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.156816    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-trusted-ca-bundle\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:09.156948 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.156826    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-console-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:09.156948 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.156835    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-oauth-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:09.156948 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.156842    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aec1bdb-9206-4093-94dc-a024b4db4239-oauth-serving-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:09.156948 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.156850    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aec1bdb-9206-4093-94dc-a024b4db4239-console-serving-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:09.156948 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.156858    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77xvl\" (UniqueName: \"kubernetes.io/projected/7aec1bdb-9206-4093-94dc-a024b4db4239-kube-api-access-77xvl\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:09.736213 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.736149    2580 generic.go:356] "Generic (PLEG): container finished" podID="7aec1bdb-9206-4093-94dc-a024b4db4239" containerID="e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950" exitCode=0
Sep 06 05:02:09.736213 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.736214    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc95747bc-kfzpx"
Sep 06 05:02:09.736833 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.736246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc95747bc-kfzpx" event={"ID":"7aec1bdb-9206-4093-94dc-a024b4db4239","Type":"ContainerDied","Data":"e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950"}
Sep 06 05:02:09.736833 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.736293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc95747bc-kfzpx" event={"ID":"7aec1bdb-9206-4093-94dc-a024b4db4239","Type":"ContainerDied","Data":"c7008de9ade67d912f2309ec1639755f7420ed8dfb639924c898e69b65f0089b"}
Sep 06 05:02:09.736833 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.736311    2580 scope.go:117] "RemoveContainer" containerID="e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950"
Sep 06 05:02:09.745480 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.745465    2580 scope.go:117] "RemoveContainer" containerID="e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950"
Sep 06 05:02:09.745760 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:02:09.745740    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950\": container with ID starting with e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950 not found: ID does not exist" containerID="e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950"
Sep 06 05:02:09.745796 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.745769    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950"} err="failed to get container status \"e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950\": rpc error: code = NotFound desc = could not find container \"e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950\": container with ID starting with e7958309b3b74d54caa3f9bed9bbe8be15460c36bb9ba14e410e97facfc13950 not found: ID does not exist"
Sep 06 05:02:09.756652 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.756631    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bc95747bc-kfzpx"]
Sep 06 05:02:09.760018 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:09.759997    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bc95747bc-kfzpx"]
Sep 06 05:02:10.346379 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:10.346350    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aec1bdb-9206-4093-94dc-a024b4db4239" path="/var/lib/kubelet/pods/7aec1bdb-9206-4093-94dc-a024b4db4239/volumes"
Sep 06 05:02:13.241611 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.241574    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"]
Sep 06 05:02:13.241991 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.241892    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" containerName="registry"
Sep 06 05:02:13.241991 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.241906    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" containerName="registry"
Sep 06 05:02:13.241991 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.241929    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aec1bdb-9206-4093-94dc-a024b4db4239" containerName="console"
Sep 06 05:02:13.241991 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.241934    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aec1bdb-9206-4093-94dc-a024b4db4239" containerName="console"
Sep 06 05:02:13.241991 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.241984    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aec1bdb-9206-4093-94dc-a024b4db4239" containerName="console"
Sep 06 05:02:13.241991 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.241993    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="531caa8b-f2b0-41f1-9c91-b68f8fbbfa33" containerName="registry"
Sep 06 05:02:13.244721 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.244705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.246920 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.246899    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 06 05:02:13.246920 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.246924    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 06 05:02:13.247757 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.247742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-k7tq2\""
Sep 06 05:02:13.251657 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.251632    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"]
Sep 06 05:02:13.283913 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.283894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.284013 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.283953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.284013 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.283984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtxc2\" (UniqueName: \"kubernetes.io/projected/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-kube-api-access-mtxc2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.384432 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.384396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.384432 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.384437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtxc2\" (UniqueName: \"kubernetes.io/projected/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-kube-api-access-mtxc2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.384670 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.384469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.384765 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.384744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.384837 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.384815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.392513 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.392489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtxc2\" (UniqueName: \"kubernetes.io/projected/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-kube-api-access-mtxc2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.554549 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.554487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:13.821833 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:13.821808    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"]
Sep 06 05:02:13.824463 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:02:13.824440    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ddb10b6_df71_4efc_b190_d74b8d1bfdbb.slice/crio-0107843584af1595f78fc4b309844569893b77f88a3f30215736c76602a73bab WatchSource:0}: Error finding container 0107843584af1595f78fc4b309844569893b77f88a3f30215736c76602a73bab: Status 404 returned error can't find the container with id 0107843584af1595f78fc4b309844569893b77f88a3f30215736c76602a73bab
Sep 06 05:02:14.760461 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:14.760414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm" event={"ID":"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb","Type":"ContainerStarted","Data":"0107843584af1595f78fc4b309844569893b77f88a3f30215736c76602a73bab"}
Sep 06 05:02:20.787895 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:20.787859    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" containerID="7cfb1d2c5004a3bc5763e43df876ed4d1d9b713701f77a3aa9303f1d7e02b5f8" exitCode=0
Sep 06 05:02:20.787895 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:20.787898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm" event={"ID":"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb","Type":"ContainerDied","Data":"7cfb1d2c5004a3bc5763e43df876ed4d1d9b713701f77a3aa9303f1d7e02b5f8"}
Sep 06 05:02:23.802391 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:23.802351    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" containerID="705ddbfe575b1d0833a68f7f33f963d58e95236fa3800e59e9c1f6e312803b0d" exitCode=0
Sep 06 05:02:23.802755 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:23.802406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm" event={"ID":"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb","Type":"ContainerDied","Data":"705ddbfe575b1d0833a68f7f33f963d58e95236fa3800e59e9c1f6e312803b0d"}
Sep 06 05:02:31.840124 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:31.840090    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" containerID="11f9bf1aac09117d5a4004937995d30b492d044f5863f95f625eba3bc4073eb8" exitCode=0
Sep 06 05:02:31.840527 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:31.840190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm" event={"ID":"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb","Type":"ContainerDied","Data":"11f9bf1aac09117d5a4004937995d30b492d044f5863f95f625eba3bc4073eb8"}
Sep 06 05:02:32.963844 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:32.963821    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:33.053568 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.053539    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtxc2\" (UniqueName: \"kubernetes.io/projected/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-kube-api-access-mtxc2\") pod \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") "
Sep 06 05:02:33.053742 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.053595    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-bundle\") pod \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") "
Sep 06 05:02:33.053742 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.053671    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-util\") pod \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\" (UID: \"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb\") "
Sep 06 05:02:33.054303 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.054273    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-bundle" (OuterVolumeSpecName: "bundle") pod "9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" (UID: "9ddb10b6-df71-4efc-b190-d74b8d1bfdbb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:02:33.055848 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.055820    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-kube-api-access-mtxc2" (OuterVolumeSpecName: "kube-api-access-mtxc2") pod "9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" (UID: "9ddb10b6-df71-4efc-b190-d74b8d1bfdbb"). InnerVolumeSpecName "kube-api-access-mtxc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:02:33.058253 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.058231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-util" (OuterVolumeSpecName: "util") pod "9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" (UID: "9ddb10b6-df71-4efc-b190-d74b8d1bfdbb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:02:33.154372 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.154312    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-util\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:33.154372 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.154335    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtxc2\" (UniqueName: \"kubernetes.io/projected/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-kube-api-access-mtxc2\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:33.154372 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.154348    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ddb10b6-df71-4efc-b190-d74b8d1bfdbb-bundle\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:33.849640 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.849605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm" event={"ID":"9ddb10b6-df71-4efc-b190-d74b8d1bfdbb","Type":"ContainerDied","Data":"0107843584af1595f78fc4b309844569893b77f88a3f30215736c76602a73bab"}
Sep 06 05:02:33.849640 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.849628    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lx4nm"
Sep 06 05:02:33.849831 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:33.849683    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0107843584af1595f78fc4b309844569893b77f88a3f30215736c76602a73bab"
Sep 06 05:02:41.142632 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.142593    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"]
Sep 06 05:02:41.143099 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.142995    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" containerName="extract"
Sep 06 05:02:41.143099 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.143009    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" containerName="extract"
Sep 06 05:02:41.143099 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.143018    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" containerName="util"
Sep 06 05:02:41.143099 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.143024    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" containerName="util"
Sep 06 05:02:41.143099 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.143040    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" containerName="pull"
Sep 06 05:02:41.143099 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.143045    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" containerName="pull"
Sep 06 05:02:41.143099 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.143093    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ddb10b6-df71-4efc-b190-d74b8d1bfdbb" containerName="extract"
Sep 06 05:02:41.146512 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.146498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"
Sep 06 05:02:41.149455 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.149428    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-q877r\""
Sep 06 05:02:41.149577 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.149456    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 06 05:02:41.149577 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.149556    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 06 05:02:41.168878 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.168856    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"]
Sep 06 05:02:41.318676 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.318652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq5n4\" (UniqueName: \"kubernetes.io/projected/6dd215e7-d704-4a5b-aaf2-3e7711043f1e-kube-api-access-rq5n4\") pod \"cert-manager-operator-controller-manager-55c444bf86-h7rz9\" (UID: \"6dd215e7-d704-4a5b-aaf2-3e7711043f1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"
Sep 06 05:02:41.318797 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.318695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dd215e7-d704-4a5b-aaf2-3e7711043f1e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-h7rz9\" (UID: \"6dd215e7-d704-4a5b-aaf2-3e7711043f1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"
Sep 06 05:02:41.419923 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.419865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq5n4\" (UniqueName: \"kubernetes.io/projected/6dd215e7-d704-4a5b-aaf2-3e7711043f1e-kube-api-access-rq5n4\") pod \"cert-manager-operator-controller-manager-55c444bf86-h7rz9\" (UID: \"6dd215e7-d704-4a5b-aaf2-3e7711043f1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"
Sep 06 05:02:41.419923 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.419899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dd215e7-d704-4a5b-aaf2-3e7711043f1e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-h7rz9\" (UID: \"6dd215e7-d704-4a5b-aaf2-3e7711043f1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"
Sep 06 05:02:41.420308 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.420291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dd215e7-d704-4a5b-aaf2-3e7711043f1e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-h7rz9\" (UID: \"6dd215e7-d704-4a5b-aaf2-3e7711043f1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"
Sep 06 05:02:41.431060 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.431028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq5n4\" (UniqueName: \"kubernetes.io/projected/6dd215e7-d704-4a5b-aaf2-3e7711043f1e-kube-api-access-rq5n4\") pod \"cert-manager-operator-controller-manager-55c444bf86-h7rz9\" (UID: \"6dd215e7-d704-4a5b-aaf2-3e7711043f1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"
Sep 06 05:02:41.455829 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.455808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"
Sep 06 05:02:41.739437 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.739405    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9"]
Sep 06 05:02:41.741120 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:02:41.741096    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd215e7_d704_4a5b_aaf2_3e7711043f1e.slice/crio-dfff609aa5bd2263011bdc340fbd0dbbf66081048c77096f709a2e35bc551993 WatchSource:0}: Error finding container dfff609aa5bd2263011bdc340fbd0dbbf66081048c77096f709a2e35bc551993: Status 404 returned error can't find the container with id dfff609aa5bd2263011bdc340fbd0dbbf66081048c77096f709a2e35bc551993
Sep 06 05:02:41.883494 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:41.883463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9" event={"ID":"6dd215e7-d704-4a5b-aaf2-3e7711043f1e","Type":"ContainerStarted","Data":"dfff609aa5bd2263011bdc340fbd0dbbf66081048c77096f709a2e35bc551993"}
Sep 06 05:02:43.896170 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:43.896117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9" event={"ID":"6dd215e7-d704-4a5b-aaf2-3e7711043f1e","Type":"ContainerStarted","Data":"41a000a0331d3fac2f3289c7bb59feef9c864caca65b2b10b16e89c0af2dfa01"}
Sep 06 05:02:43.915616 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:43.915570    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h7rz9" podStartSLOduration=1.340615992 podStartE2EDuration="2.915550814s" totalImagesPullingTime="1.574934822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:02:43.914024852 +0000 UTC m=+440.055584811" watchObservedRunningTime="2026-09-06 05:02:43.915550814 +0000 UTC m=+440.057110758"
Sep 06 05:02:44.876387 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:44.876354    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"]
Sep 06 05:02:44.878717 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:44.878699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:44.881094 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:44.881057    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-k7tq2\""
Sep 06 05:02:44.881094 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:44.881085    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 06 05:02:44.882200 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:44.882182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 06 05:02:44.887509 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:44.887485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"]
Sep 06 05:02:45.052531 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.052499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:45.052531 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.052534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:45.052895 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.052566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867m5\" (UniqueName: \"kubernetes.io/projected/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-kube-api-access-867m5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:45.153840 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.153779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:45.153840 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.153807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:45.153840 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.153832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867m5\" (UniqueName: \"kubernetes.io/projected/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-kube-api-access-867m5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:45.154122 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.154102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:45.154188 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.154138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:45.164451 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.164428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-867m5\" (UniqueName: \"kubernetes.io/projected/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-kube-api-access-867m5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:45.188355 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.188333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:45.461517 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.461491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"]
Sep 06 05:02:45.464130 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:02:45.464103    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d5681c_5d6c_4beb_8bbe_2e08d166f97e.slice/crio-d2d9c7cbe880fe8ae0d9a3e60173db4c916c8e3173a8efcc27645d36ee6d1933 WatchSource:0}: Error finding container d2d9c7cbe880fe8ae0d9a3e60173db4c916c8e3173a8efcc27645d36ee6d1933: Status 404 returned error can't find the container with id d2d9c7cbe880fe8ae0d9a3e60173db4c916c8e3173a8efcc27645d36ee6d1933
Sep 06 05:02:45.905809 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.905729    2580 generic.go:356] "Generic (PLEG): container finished" podID="44d5681c-5d6c-4beb-8bbe-2e08d166f97e" containerID="35dc23d3e0ea8eb62bf4d39a08da16a518a7cc9d157106dbb741e36d7acf2706" exitCode=0
Sep 06 05:02:45.906049 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.905810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw" event={"ID":"44d5681c-5d6c-4beb-8bbe-2e08d166f97e","Type":"ContainerDied","Data":"35dc23d3e0ea8eb62bf4d39a08da16a518a7cc9d157106dbb741e36d7acf2706"}
Sep 06 05:02:45.906049 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:45.905856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw" event={"ID":"44d5681c-5d6c-4beb-8bbe-2e08d166f97e","Type":"ContainerStarted","Data":"d2d9c7cbe880fe8ae0d9a3e60173db4c916c8e3173a8efcc27645d36ee6d1933"}
Sep 06 05:02:47.867827 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:47.867799    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"]
Sep 06 05:02:47.870530 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:47.870510    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"
Sep 06 05:02:47.872646 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:47.872625    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 06 05:02:47.872726 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:47.872638    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 06 05:02:47.873427 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:47.873411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sqbvb\""
Sep 06 05:02:47.881779 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:47.881759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"]
Sep 06 05:02:47.977681 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:47.977645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44mq\" (UniqueName: \"kubernetes.io/projected/7158c47a-c256-45a2-8b6d-4c783393d42b-kube-api-access-g44mq\") pod \"cert-manager-cainjector-559cdd8c87-qpgf6\" (UID: \"7158c47a-c256-45a2-8b6d-4c783393d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"
Sep 06 05:02:47.977812 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:47.977759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7158c47a-c256-45a2-8b6d-4c783393d42b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qpgf6\" (UID: \"7158c47a-c256-45a2-8b6d-4c783393d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"
Sep 06 05:02:48.079009 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:48.078979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44mq\" (UniqueName: \"kubernetes.io/projected/7158c47a-c256-45a2-8b6d-4c783393d42b-kube-api-access-g44mq\") pod \"cert-manager-cainjector-559cdd8c87-qpgf6\" (UID: \"7158c47a-c256-45a2-8b6d-4c783393d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"
Sep 06 05:02:48.079183 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:48.079050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7158c47a-c256-45a2-8b6d-4c783393d42b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qpgf6\" (UID: \"7158c47a-c256-45a2-8b6d-4c783393d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"
Sep 06 05:02:48.086349 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:48.086326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7158c47a-c256-45a2-8b6d-4c783393d42b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qpgf6\" (UID: \"7158c47a-c256-45a2-8b6d-4c783393d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"
Sep 06 05:02:48.086521 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:48.086491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44mq\" (UniqueName: \"kubernetes.io/projected/7158c47a-c256-45a2-8b6d-4c783393d42b-kube-api-access-g44mq\") pod \"cert-manager-cainjector-559cdd8c87-qpgf6\" (UID: \"7158c47a-c256-45a2-8b6d-4c783393d42b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"
Sep 06 05:02:48.202974 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:48.202915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"
Sep 06 05:02:48.471858 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:48.471722    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6"]
Sep 06 05:02:48.474360 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:02:48.474340    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7158c47a_c256_45a2_8b6d_4c783393d42b.slice/crio-541d8b4934b86fe7cb4b03c3b47554b7d4a936c8a71b91956a295de7ae581ab3 WatchSource:0}: Error finding container 541d8b4934b86fe7cb4b03c3b47554b7d4a936c8a71b91956a295de7ae581ab3: Status 404 returned error can't find the container with id 541d8b4934b86fe7cb4b03c3b47554b7d4a936c8a71b91956a295de7ae581ab3
Sep 06 05:02:48.921610 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:48.921536    2580 generic.go:356] "Generic (PLEG): container finished" podID="44d5681c-5d6c-4beb-8bbe-2e08d166f97e" containerID="1de843e78cfd3785d699f7ac9d1b989581dd4f9951398dc518bd2ead26a15939" exitCode=0
Sep 06 05:02:48.921991 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:48.921623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw" event={"ID":"44d5681c-5d6c-4beb-8bbe-2e08d166f97e","Type":"ContainerDied","Data":"1de843e78cfd3785d699f7ac9d1b989581dd4f9951398dc518bd2ead26a15939"}
Sep 06 05:02:48.922866 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:48.922843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6" event={"ID":"7158c47a-c256-45a2-8b6d-4c783393d42b","Type":"ContainerStarted","Data":"541d8b4934b86fe7cb4b03c3b47554b7d4a936c8a71b91956a295de7ae581ab3"}
Sep 06 05:02:49.930320 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:49.930280    2580 generic.go:356] "Generic (PLEG): container finished" podID="44d5681c-5d6c-4beb-8bbe-2e08d166f97e" containerID="1e181368541e6c0b82f9e9e61d1a3fbf56b83c65e67a66c85b97b079ed527860" exitCode=0
Sep 06 05:02:49.930740 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:49.930343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw" event={"ID":"44d5681c-5d6c-4beb-8bbe-2e08d166f97e","Type":"ContainerDied","Data":"1e181368541e6c0b82f9e9e61d1a3fbf56b83c65e67a66c85b97b079ed527860"}
Sep 06 05:02:51.070783 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.070760    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:51.103438 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.103404    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-867m5\" (UniqueName: \"kubernetes.io/projected/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-kube-api-access-867m5\") pod \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") "
Sep 06 05:02:51.103550 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.103486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-bundle\") pod \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") "
Sep 06 05:02:51.103550 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.103529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-util\") pod \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\" (UID: \"44d5681c-5d6c-4beb-8bbe-2e08d166f97e\") "
Sep 06 05:02:51.103940 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.103915    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-bundle" (OuterVolumeSpecName: "bundle") pod "44d5681c-5d6c-4beb-8bbe-2e08d166f97e" (UID: "44d5681c-5d6c-4beb-8bbe-2e08d166f97e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:02:51.105751 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.105723    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-kube-api-access-867m5" (OuterVolumeSpecName: "kube-api-access-867m5") pod "44d5681c-5d6c-4beb-8bbe-2e08d166f97e" (UID: "44d5681c-5d6c-4beb-8bbe-2e08d166f97e"). InnerVolumeSpecName "kube-api-access-867m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:02:51.110672 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.110584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-util" (OuterVolumeSpecName: "util") pod "44d5681c-5d6c-4beb-8bbe-2e08d166f97e" (UID: "44d5681c-5d6c-4beb-8bbe-2e08d166f97e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:02:51.204411 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.204334    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-util\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:51.204411 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.204370    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-867m5\" (UniqueName: \"kubernetes.io/projected/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-kube-api-access-867m5\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:51.204411 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.204380    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44d5681c-5d6c-4beb-8bbe-2e08d166f97e-bundle\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:02:51.941663 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.941625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw" event={"ID":"44d5681c-5d6c-4beb-8bbe-2e08d166f97e","Type":"ContainerDied","Data":"d2d9c7cbe880fe8ae0d9a3e60173db4c916c8e3173a8efcc27645d36ee6d1933"}
Sep 06 05:02:51.941858 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.941678    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d9c7cbe880fe8ae0d9a3e60173db4c916c8e3173a8efcc27645d36ee6d1933"
Sep 06 05:02:51.941858 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.941631    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr75tw"
Sep 06 05:02:51.945752 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.945719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6" event={"ID":"7158c47a-c256-45a2-8b6d-4c783393d42b","Type":"ContainerStarted","Data":"61b862d245805496f841dacc0d5aea018f7b98fb6c52f7d77a754fc47113c125"}
Sep 06 05:02:51.959241 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:51.959199    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpgf6" podStartSLOduration=2.457510796 podStartE2EDuration="4.959186409s" totalImagesPullingTime="2.501675613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:02:51.958770148 +0000 UTC m=+448.100330090" watchObservedRunningTime="2026-09-06 05:02:51.959186409 +0000 UTC m=+448.100746347"
Sep 06 05:02:53.457744 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.457710    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xr4t4"]
Sep 06 05:02:53.458123 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.458004    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d5681c-5d6c-4beb-8bbe-2e08d166f97e" containerName="extract"
Sep 06 05:02:53.458123 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.458015    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d5681c-5d6c-4beb-8bbe-2e08d166f97e" containerName="extract"
Sep 06 05:02:53.458123 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.458034    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d5681c-5d6c-4beb-8bbe-2e08d166f97e" containerName="util"
Sep 06 05:02:53.458123 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.458040    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d5681c-5d6c-4beb-8bbe-2e08d166f97e" containerName="util"
Sep 06 05:02:53.458123 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.458053    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d5681c-5d6c-4beb-8bbe-2e08d166f97e" containerName="pull"
Sep 06 05:02:53.458123 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.458059    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d5681c-5d6c-4beb-8bbe-2e08d166f97e" containerName="pull"
Sep 06 05:02:53.458123 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.458104    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d5681c-5d6c-4beb-8bbe-2e08d166f97e" containerName="extract"
Sep 06 05:02:53.459985 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.459965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4"
Sep 06 05:02:53.461991 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.461969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-t7dhl\""
Sep 06 05:02:53.468845 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.468826    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xr4t4"]
Sep 06 05:02:53.518312 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.518284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6dmg\" (UniqueName: \"kubernetes.io/projected/38d15621-f8cc-4d88-ace4-1d3793879b0a-kube-api-access-v6dmg\") pod \"cert-manager-webhook-6486ffd997-xr4t4\" (UID: \"38d15621-f8cc-4d88-ace4-1d3793879b0a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4"
Sep 06 05:02:53.518421 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.518343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38d15621-f8cc-4d88-ace4-1d3793879b0a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xr4t4\" (UID: \"38d15621-f8cc-4d88-ace4-1d3793879b0a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4"
Sep 06 05:02:53.619055 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.619027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6dmg\" (UniqueName: \"kubernetes.io/projected/38d15621-f8cc-4d88-ace4-1d3793879b0a-kube-api-access-v6dmg\") pod \"cert-manager-webhook-6486ffd997-xr4t4\" (UID: \"38d15621-f8cc-4d88-ace4-1d3793879b0a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4"
Sep 06 05:02:53.619202 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.619080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38d15621-f8cc-4d88-ace4-1d3793879b0a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xr4t4\" (UID: \"38d15621-f8cc-4d88-ace4-1d3793879b0a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4"
Sep 06 05:02:53.631287 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.631261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6dmg\" (UniqueName: \"kubernetes.io/projected/38d15621-f8cc-4d88-ace4-1d3793879b0a-kube-api-access-v6dmg\") pod \"cert-manager-webhook-6486ffd997-xr4t4\" (UID: \"38d15621-f8cc-4d88-ace4-1d3793879b0a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4"
Sep 06 05:02:53.640187 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.640148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38d15621-f8cc-4d88-ace4-1d3793879b0a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xr4t4\" (UID: \"38d15621-f8cc-4d88-ace4-1d3793879b0a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4"
Sep 06 05:02:53.769204 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:53.769176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4"
Sep 06 05:02:54.035361 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:54.035339    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xr4t4"]
Sep 06 05:02:54.037381 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:02:54.037355    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38d15621_f8cc_4d88_ace4_1d3793879b0a.slice/crio-e7052d6fb32d91b37eb0a703890a8f21a0fb44ab628ea6a49b59b2a9420378a0 WatchSource:0}: Error finding container e7052d6fb32d91b37eb0a703890a8f21a0fb44ab628ea6a49b59b2a9420378a0: Status 404 returned error can't find the container with id e7052d6fb32d91b37eb0a703890a8f21a0fb44ab628ea6a49b59b2a9420378a0
Sep 06 05:02:54.960527 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:54.960495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4" event={"ID":"38d15621-f8cc-4d88-ace4-1d3793879b0a","Type":"ContainerStarted","Data":"d34c9cf970efa1788af4f95f77c47baafaaf3f12978f60661e941f8c5e38e987"}
Sep 06 05:02:54.960889 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:54.960535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4" event={"ID":"38d15621-f8cc-4d88-ace4-1d3793879b0a","Type":"ContainerStarted","Data":"e7052d6fb32d91b37eb0a703890a8f21a0fb44ab628ea6a49b59b2a9420378a0"}
Sep 06 05:02:54.960889 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:54.960572    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4"
Sep 06 05:02:54.975763 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:54.975718    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4" podStartSLOduration=1.9757037149999999 podStartE2EDuration="1.975703715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:02:54.974205794 +0000 UTC m=+451.115765735" watchObservedRunningTime="2026-09-06 05:02:54.975703715 +0000 UTC m=+451.117263658"
Sep 06 05:02:56.521227 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.521196    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5n8s8"]
Sep 06 05:02:56.527478 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.527460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5n8s8"
Sep 06 05:02:56.529530 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.529508    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6dbp6\""
Sep 06 05:02:56.533441 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.533417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5n8s8"]
Sep 06 05:02:56.537571 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.537548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/957c0545-278c-41a0-83d9-3554c4d75c93-bound-sa-token\") pod \"cert-manager-55b4d954d5-5n8s8\" (UID: \"957c0545-278c-41a0-83d9-3554c4d75c93\") " pod="cert-manager/cert-manager-55b4d954d5-5n8s8"
Sep 06 05:02:56.537670 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.537582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckdr\" (UniqueName: \"kubernetes.io/projected/957c0545-278c-41a0-83d9-3554c4d75c93-kube-api-access-2ckdr\") pod \"cert-manager-55b4d954d5-5n8s8\" (UID: \"957c0545-278c-41a0-83d9-3554c4d75c93\") " pod="cert-manager/cert-manager-55b4d954d5-5n8s8"
Sep 06 05:02:56.638879 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.638856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/957c0545-278c-41a0-83d9-3554c4d75c93-bound-sa-token\") pod \"cert-manager-55b4d954d5-5n8s8\" (UID: \"957c0545-278c-41a0-83d9-3554c4d75c93\") " pod="cert-manager/cert-manager-55b4d954d5-5n8s8"
Sep 06 05:02:56.639000 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.638881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckdr\" (UniqueName: \"kubernetes.io/projected/957c0545-278c-41a0-83d9-3554c4d75c93-kube-api-access-2ckdr\") pod \"cert-manager-55b4d954d5-5n8s8\" (UID: \"957c0545-278c-41a0-83d9-3554c4d75c93\") " pod="cert-manager/cert-manager-55b4d954d5-5n8s8"
Sep 06 05:02:56.655125 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.655096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/957c0545-278c-41a0-83d9-3554c4d75c93-bound-sa-token\") pod \"cert-manager-55b4d954d5-5n8s8\" (UID: \"957c0545-278c-41a0-83d9-3554c4d75c93\") " pod="cert-manager/cert-manager-55b4d954d5-5n8s8"
Sep 06 05:02:56.655305 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.655286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckdr\" (UniqueName: \"kubernetes.io/projected/957c0545-278c-41a0-83d9-3554c4d75c93-kube-api-access-2ckdr\") pod \"cert-manager-55b4d954d5-5n8s8\" (UID: \"957c0545-278c-41a0-83d9-3554c4d75c93\") " pod="cert-manager/cert-manager-55b4d954d5-5n8s8"
Sep 06 05:02:56.838363 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:56.838299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5n8s8"
Sep 06 05:02:57.110515 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:57.110449    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5n8s8"]
Sep 06 05:02:57.112644 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:02:57.112614    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod957c0545_278c_41a0_83d9_3554c4d75c93.slice/crio-00870c7b020bf52c893ac2a4a87a366b0ee5c1b98fde4b5a0e13222cf57a8d01 WatchSource:0}: Error finding container 00870c7b020bf52c893ac2a4a87a366b0ee5c1b98fde4b5a0e13222cf57a8d01: Status 404 returned error can't find the container with id 00870c7b020bf52c893ac2a4a87a366b0ee5c1b98fde4b5a0e13222cf57a8d01
Sep 06 05:02:57.975506 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:57.975473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5n8s8" event={"ID":"957c0545-278c-41a0-83d9-3554c4d75c93","Type":"ContainerStarted","Data":"504fac35d2e1fa1b773c79a78d07e7cf91c4baf54c5d319db25ae6bb858497d0"}
Sep 06 05:02:57.975886 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:57.975513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5n8s8" event={"ID":"957c0545-278c-41a0-83d9-3554c4d75c93","Type":"ContainerStarted","Data":"00870c7b020bf52c893ac2a4a87a366b0ee5c1b98fde4b5a0e13222cf57a8d01"}
Sep 06 05:02:57.990135 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:02:57.990091    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-5n8s8" podStartSLOduration=1.990076857 podStartE2EDuration="1.990076857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:02:57.988701318 +0000 UTC m=+454.130261261" watchObservedRunningTime="2026-09-06 05:02:57.990076857 +0000 UTC m=+454.131636798"
Sep 06 05:03:00.968025 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:00.967993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xr4t4"
Sep 06 05:03:05.449480 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.449445    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"]
Sep 06 05:03:05.454406 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.454386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:05.456674 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.456651    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 06 05:03:05.457727 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.457703    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-k7tq2\""
Sep 06 05:03:05.457975 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.457947    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 06 05:03:05.459965 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.459943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"]
Sep 06 05:03:05.505836 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.505810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl7td\" (UniqueName: \"kubernetes.io/projected/9f79ff94-522a-482e-9d8d-cd77738ffc11-kube-api-access-kl7td\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:05.505938 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.505846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:05.505938 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.505871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:05.606235 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.606211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl7td\" (UniqueName: \"kubernetes.io/projected/9f79ff94-522a-482e-9d8d-cd77738ffc11-kube-api-access-kl7td\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:05.606336 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.606246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:05.606336 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.606270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:05.606573 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.606554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:05.606627 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.606581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:05.614145 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.614124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl7td\" (UniqueName: \"kubernetes.io/projected/9f79ff94-522a-482e-9d8d-cd77738ffc11-kube-api-access-kl7td\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:05.766301 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:05.766270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:06.035601 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:06.035578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"]
Sep 06 05:03:06.037723 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:03:06.037695    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f79ff94_522a_482e_9d8d_cd77738ffc11.slice/crio-2feada34f6f4444b4ffd035a4aa2640be3224ab338cb4ed0a69382de4aa1dd85 WatchSource:0}: Error finding container 2feada34f6f4444b4ffd035a4aa2640be3224ab338cb4ed0a69382de4aa1dd85: Status 404 returned error can't find the container with id 2feada34f6f4444b4ffd035a4aa2640be3224ab338cb4ed0a69382de4aa1dd85
Sep 06 05:03:07.021720 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:07.021685    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f79ff94-522a-482e-9d8d-cd77738ffc11" containerID="b3ea92285a3b28ca8893573a03cf4ab57c9035b4767e3c3a819524ed5f0a3f72" exitCode=0
Sep 06 05:03:07.022147 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:07.021745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt" event={"ID":"9f79ff94-522a-482e-9d8d-cd77738ffc11","Type":"ContainerDied","Data":"b3ea92285a3b28ca8893573a03cf4ab57c9035b4767e3c3a819524ed5f0a3f72"}
Sep 06 05:03:07.022147 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:07.021793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt" event={"ID":"9f79ff94-522a-482e-9d8d-cd77738ffc11","Type":"ContainerStarted","Data":"2feada34f6f4444b4ffd035a4aa2640be3224ab338cb4ed0a69382de4aa1dd85"}
Sep 06 05:03:08.027686 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:08.027652    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f79ff94-522a-482e-9d8d-cd77738ffc11" containerID="8830e48cc391602dc61332a140a2da25a7c9292721d6bb450fb353cf2da00b20" exitCode=0
Sep 06 05:03:08.027997 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:08.027697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt" event={"ID":"9f79ff94-522a-482e-9d8d-cd77738ffc11","Type":"ContainerDied","Data":"8830e48cc391602dc61332a140a2da25a7c9292721d6bb450fb353cf2da00b20"}
Sep 06 05:03:09.033956 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:09.033920    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f79ff94-522a-482e-9d8d-cd77738ffc11" containerID="a4dfdfc5b24a1d8642bd4775ff4295beca4e97db78b87fa5c9312d3cde110a2e" exitCode=0
Sep 06 05:03:09.034338 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:09.034000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt" event={"ID":"9f79ff94-522a-482e-9d8d-cd77738ffc11","Type":"ContainerDied","Data":"a4dfdfc5b24a1d8642bd4775ff4295beca4e97db78b87fa5c9312d3cde110a2e"}
Sep 06 05:03:10.154609 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:10.154586    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:10.238952 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:10.238908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-util\") pod \"9f79ff94-522a-482e-9d8d-cd77738ffc11\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") "
Sep 06 05:03:10.239120 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:10.239005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl7td\" (UniqueName: \"kubernetes.io/projected/9f79ff94-522a-482e-9d8d-cd77738ffc11-kube-api-access-kl7td\") pod \"9f79ff94-522a-482e-9d8d-cd77738ffc11\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") "
Sep 06 05:03:10.239120 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:10.239035    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-bundle\") pod \"9f79ff94-522a-482e-9d8d-cd77738ffc11\" (UID: \"9f79ff94-522a-482e-9d8d-cd77738ffc11\") "
Sep 06 05:03:10.239804 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:10.239777    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-bundle" (OuterVolumeSpecName: "bundle") pod "9f79ff94-522a-482e-9d8d-cd77738ffc11" (UID: "9f79ff94-522a-482e-9d8d-cd77738ffc11"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:03:10.241223 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:10.241201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f79ff94-522a-482e-9d8d-cd77738ffc11-kube-api-access-kl7td" (OuterVolumeSpecName: "kube-api-access-kl7td") pod "9f79ff94-522a-482e-9d8d-cd77738ffc11" (UID: "9f79ff94-522a-482e-9d8d-cd77738ffc11"). InnerVolumeSpecName "kube-api-access-kl7td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:03:10.244042 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:10.244019    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-util" (OuterVolumeSpecName: "util") pod "9f79ff94-522a-482e-9d8d-cd77738ffc11" (UID: "9f79ff94-522a-482e-9d8d-cd77738ffc11"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:03:10.339659 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:10.339603    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl7td\" (UniqueName: \"kubernetes.io/projected/9f79ff94-522a-482e-9d8d-cd77738ffc11-kube-api-access-kl7td\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:03:10.339659 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:10.339624    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-bundle\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:03:10.339659 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:10.339633    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f79ff94-522a-482e-9d8d-cd77738ffc11-util\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:03:11.045826 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:11.045792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt" event={"ID":"9f79ff94-522a-482e-9d8d-cd77738ffc11","Type":"ContainerDied","Data":"2feada34f6f4444b4ffd035a4aa2640be3224ab338cb4ed0a69382de4aa1dd85"}
Sep 06 05:03:11.046001 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:11.045842    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2feada34f6f4444b4ffd035a4aa2640be3224ab338cb4ed0a69382de4aa1dd85"
Sep 06 05:03:11.046001 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:11.045856    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5kqpt"
Sep 06 05:03:15.888664 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.888630    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"]
Sep 06 05:03:15.889038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.888950    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f79ff94-522a-482e-9d8d-cd77738ffc11" containerName="extract"
Sep 06 05:03:15.889038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.888960    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f79ff94-522a-482e-9d8d-cd77738ffc11" containerName="extract"
Sep 06 05:03:15.889038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.888973    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f79ff94-522a-482e-9d8d-cd77738ffc11" containerName="util"
Sep 06 05:03:15.889038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.888978    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f79ff94-522a-482e-9d8d-cd77738ffc11" containerName="util"
Sep 06 05:03:15.889038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.888994    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f79ff94-522a-482e-9d8d-cd77738ffc11" containerName="pull"
Sep 06 05:03:15.889038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.889010    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f79ff94-522a-482e-9d8d-cd77738ffc11" containerName="pull"
Sep 06 05:03:15.889245 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.889061    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f79ff94-522a-482e-9d8d-cd77738ffc11" containerName="extract"
Sep 06 05:03:15.891910 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.891883    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:15.894257 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.894237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 06 05:03:15.895047 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.895029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 06 05:03:15.895126 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.895035    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-k7tq2\""
Sep 06 05:03:15.902689 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.902664    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"]
Sep 06 05:03:15.982024 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.981997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:15.982024 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.982024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:15.982196 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:15.982053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92mfq\" (UniqueName: \"kubernetes.io/projected/80b516ba-02ab-4c55-aef0-266c6d9a2767-kube-api-access-92mfq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:16.083059 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:16.083030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:16.083193 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:16.083060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:16.083193 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:16.083100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92mfq\" (UniqueName: \"kubernetes.io/projected/80b516ba-02ab-4c55-aef0-266c6d9a2767-kube-api-access-92mfq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:16.083386 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:16.083369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:16.083443 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:16.083401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:16.096721 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:16.096697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92mfq\" (UniqueName: \"kubernetes.io/projected/80b516ba-02ab-4c55-aef0-266c6d9a2767-kube-api-access-92mfq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:16.201744 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:16.201685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:16.473941 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:16.473918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"]
Sep 06 05:03:16.475804 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:03:16.475777    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b516ba_02ab_4c55_aef0_266c6d9a2767.slice/crio-bec8d501425dc5f8c92ced1eb84efc300963e1ead50bc766ee0734a3a609654a WatchSource:0}: Error finding container bec8d501425dc5f8c92ced1eb84efc300963e1ead50bc766ee0734a3a609654a: Status 404 returned error can't find the container with id bec8d501425dc5f8c92ced1eb84efc300963e1ead50bc766ee0734a3a609654a
Sep 06 05:03:17.078015 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.077980    2580 generic.go:356] "Generic (PLEG): container finished" podID="80b516ba-02ab-4c55-aef0-266c6d9a2767" containerID="123bf72ab5b796e5985435ad2abf132935a714b7ae3f91ca4d06e02a3f45bf1e" exitCode=0
Sep 06 05:03:17.078525 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.078057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9" event={"ID":"80b516ba-02ab-4c55-aef0-266c6d9a2767","Type":"ContainerDied","Data":"123bf72ab5b796e5985435ad2abf132935a714b7ae3f91ca4d06e02a3f45bf1e"}
Sep 06 05:03:17.078525 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.078126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9" event={"ID":"80b516ba-02ab-4c55-aef0-266c6d9a2767","Type":"ContainerStarted","Data":"bec8d501425dc5f8c92ced1eb84efc300963e1ead50bc766ee0734a3a609654a"}
Sep 06 05:03:17.406910 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.406821    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"]
Sep 06 05:03:17.409428 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.409267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.412908 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.412885    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 06 05:03:17.413727 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.413384    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 06 05:03:17.413989 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.413429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 06 05:03:17.414429 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.413468    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p86vv\""
Sep 06 05:03:17.414429 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.413546    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 06 05:03:17.426715 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.426694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"]
Sep 06 05:03:17.491961 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.491936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-tmp\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.492060 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.491982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-manifests\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.492111 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.492073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-webhook-cert\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.492111 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.492100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.492234 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.492121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf82r\" (UniqueName: \"kubernetes.io/projected/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-kube-api-access-mf82r\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.592895 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.592862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-manifests\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.593049 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.592923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-webhook-cert\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.593049 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.592942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.593049 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.592960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf82r\" (UniqueName: \"kubernetes.io/projected/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-kube-api-access-mf82r\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.593049 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.592993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-tmp\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.593345 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.593319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-manifests\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.595241 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.595217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-tmp\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.595555 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.595532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.595645 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.595540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-webhook-cert\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.600594 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.600570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf82r\" (UniqueName: \"kubernetes.io/projected/50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35-kube-api-access-mf82r\") pod \"opendatahub-operator-controller-manager-657c48fd97-ndrr9\" (UID: \"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35\") " pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.721186 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.721144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:17.994299 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:17.994278    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"]
Sep 06 05:03:17.996232 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:03:17.996197    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b1b2e0_9d6e_4a4e_a0ef_5128b8a95d35.slice/crio-2be00d0fc656d86e765177d3a7b27693ceacbadc7473afad8af0bcc22b0ed14c WatchSource:0}: Error finding container 2be00d0fc656d86e765177d3a7b27693ceacbadc7473afad8af0bcc22b0ed14c: Status 404 returned error can't find the container with id 2be00d0fc656d86e765177d3a7b27693ceacbadc7473afad8af0bcc22b0ed14c
Sep 06 05:03:18.083794 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:18.083767    2580 generic.go:356] "Generic (PLEG): container finished" podID="80b516ba-02ab-4c55-aef0-266c6d9a2767" containerID="bec46c7701a650d6c4130504b99500c91d32971ca0af36341e2ec1ca9931e3e9" exitCode=0
Sep 06 05:03:18.084150 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:18.083804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9" event={"ID":"80b516ba-02ab-4c55-aef0-266c6d9a2767","Type":"ContainerDied","Data":"bec46c7701a650d6c4130504b99500c91d32971ca0af36341e2ec1ca9931e3e9"}
Sep 06 05:03:18.085292 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:18.085272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9" event={"ID":"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35","Type":"ContainerStarted","Data":"2be00d0fc656d86e765177d3a7b27693ceacbadc7473afad8af0bcc22b0ed14c"}
Sep 06 05:03:19.100199 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:19.100139    2580 generic.go:356] "Generic (PLEG): container finished" podID="80b516ba-02ab-4c55-aef0-266c6d9a2767" containerID="229151735c25d1cb67b38e45c86976bfa53eac6af699a981a601a57a9b354a7a" exitCode=0
Sep 06 05:03:19.100657 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:19.100259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9" event={"ID":"80b516ba-02ab-4c55-aef0-266c6d9a2767","Type":"ContainerDied","Data":"229151735c25d1cb67b38e45c86976bfa53eac6af699a981a601a57a9b354a7a"}
Sep 06 05:03:20.772327 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:20.772302    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:20.823990 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:20.823963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-util\") pod \"80b516ba-02ab-4c55-aef0-266c6d9a2767\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") "
Sep 06 05:03:20.824098 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:20.824083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92mfq\" (UniqueName: \"kubernetes.io/projected/80b516ba-02ab-4c55-aef0-266c6d9a2767-kube-api-access-92mfq\") pod \"80b516ba-02ab-4c55-aef0-266c6d9a2767\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") "
Sep 06 05:03:20.824142 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:20.824129    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-bundle\") pod \"80b516ba-02ab-4c55-aef0-266c6d9a2767\" (UID: \"80b516ba-02ab-4c55-aef0-266c6d9a2767\") "
Sep 06 05:03:20.824817 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:20.824792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-bundle" (OuterVolumeSpecName: "bundle") pod "80b516ba-02ab-4c55-aef0-266c6d9a2767" (UID: "80b516ba-02ab-4c55-aef0-266c6d9a2767"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:03:20.826179 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:20.826141    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b516ba-02ab-4c55-aef0-266c6d9a2767-kube-api-access-92mfq" (OuterVolumeSpecName: "kube-api-access-92mfq") pod "80b516ba-02ab-4c55-aef0-266c6d9a2767" (UID: "80b516ba-02ab-4c55-aef0-266c6d9a2767"). InnerVolumeSpecName "kube-api-access-92mfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:03:20.830043 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:20.830021    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-util" (OuterVolumeSpecName: "util") pod "80b516ba-02ab-4c55-aef0-266c6d9a2767" (UID: "80b516ba-02ab-4c55-aef0-266c6d9a2767"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:03:20.925224 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:20.925202    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92mfq\" (UniqueName: \"kubernetes.io/projected/80b516ba-02ab-4c55-aef0-266c6d9a2767-kube-api-access-92mfq\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:03:20.925224 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:20.925223    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-bundle\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:03:20.925337 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:20.925233    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80b516ba-02ab-4c55-aef0-266c6d9a2767-util\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:03:21.113112 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:21.113046    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9"
Sep 06 05:03:21.113293 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:21.113047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bsbc9" event={"ID":"80b516ba-02ab-4c55-aef0-266c6d9a2767","Type":"ContainerDied","Data":"bec8d501425dc5f8c92ced1eb84efc300963e1ead50bc766ee0734a3a609654a"}
Sep 06 05:03:21.113293 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:21.113201    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bec8d501425dc5f8c92ced1eb84efc300963e1ead50bc766ee0734a3a609654a"
Sep 06 05:03:21.114956 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:21.114934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9" event={"ID":"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35","Type":"ContainerStarted","Data":"513f3649efbf8c0d220610e83101d65749c64bacf03bf6feaeae443a98db9ffb"}
Sep 06 05:03:22.120845 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:22.120808    2580 generic.go:356] "Generic (PLEG): container finished" podID="50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35" containerID="513f3649efbf8c0d220610e83101d65749c64bacf03bf6feaeae443a98db9ffb" exitCode=0
Sep 06 05:03:22.121218 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:22.120887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9" event={"ID":"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35","Type":"ContainerDied","Data":"513f3649efbf8c0d220610e83101d65749c64bacf03bf6feaeae443a98db9ffb"}
Sep 06 05:03:23.128012 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:23.127977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9" event={"ID":"50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35","Type":"ContainerStarted","Data":"16113e1b3ed81c046739ee3c1d2e74b9e49bb555e9755a751790d5876cfb684f"}
Sep 06 05:03:23.128388 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:23.128203    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:23.148205 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:23.148141    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9" podStartSLOduration=1.9910502970000001 podStartE2EDuration="6.148127549s" totalImagesPullingTime="3.09239296s" totalInitContainerRuntime="1.064684292s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:03:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:03:23.145724036 +0000 UTC m=+479.287283990" watchObservedRunningTime="2026-09-06 05:03:23.148127549 +0000 UTC m=+479.289687491"
Sep 06 05:03:34.135310 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:34.135280    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-657c48fd97-ndrr9"
Sep 06 05:03:35.334692 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.334661    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"]
Sep 06 05:03:35.335042 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.334978    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b516ba-02ab-4c55-aef0-266c6d9a2767" containerName="util"
Sep 06 05:03:35.335042 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.334989    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b516ba-02ab-4c55-aef0-266c6d9a2767" containerName="util"
Sep 06 05:03:35.335042 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.335002    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b516ba-02ab-4c55-aef0-266c6d9a2767" containerName="pull"
Sep 06 05:03:35.335042 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.335007    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b516ba-02ab-4c55-aef0-266c6d9a2767" containerName="pull"
Sep 06 05:03:35.335042 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.335020    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b516ba-02ab-4c55-aef0-266c6d9a2767" containerName="extract"
Sep 06 05:03:35.335042 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.335025    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b516ba-02ab-4c55-aef0-266c6d9a2767" containerName="extract"
Sep 06 05:03:35.335253 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.335079    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b516ba-02ab-4c55-aef0-266c6d9a2767" containerName="extract"
Sep 06 05:03:35.341130 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.341106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.344789 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.344754    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 06 05:03:35.344789 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.344781    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 06 05:03:35.344964 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.344782    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-k6n4f\""
Sep 06 05:03:35.344964 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.344795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 06 05:03:35.344964 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.344783    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 06 05:03:35.344964 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.344891    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 06 05:03:35.347917 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.347896    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"]
Sep 06 05:03:35.437928 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.437897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3f3c2667-c372-4159-97e5-369c98275a03-manager-config\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.438079 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.437970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f3c2667-c372-4159-97e5-369c98275a03-cert\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.438079 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.437995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp57f\" (UniqueName: \"kubernetes.io/projected/3f3c2667-c372-4159-97e5-369c98275a03-kube-api-access-tp57f\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.438079 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.438030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f3c2667-c372-4159-97e5-369c98275a03-metrics-cert\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.539248 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.539222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3f3c2667-c372-4159-97e5-369c98275a03-manager-config\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.539378 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.539270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f3c2667-c372-4159-97e5-369c98275a03-cert\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.539378 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.539289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp57f\" (UniqueName: \"kubernetes.io/projected/3f3c2667-c372-4159-97e5-369c98275a03-kube-api-access-tp57f\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.539378 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.539315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f3c2667-c372-4159-97e5-369c98275a03-metrics-cert\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.539938 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.539908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3f3c2667-c372-4159-97e5-369c98275a03-manager-config\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.541844 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.541823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f3c2667-c372-4159-97e5-369c98275a03-cert\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.541926 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.541853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f3c2667-c372-4159-97e5-369c98275a03-metrics-cert\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.554645 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.554627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp57f\" (UniqueName: \"kubernetes.io/projected/3f3c2667-c372-4159-97e5-369c98275a03-kube-api-access-tp57f\") pod \"lws-controller-manager-7d8d45985c-gsjmd\" (UID: \"3f3c2667-c372-4159-97e5-369c98275a03\") " pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.651980 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.651900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:35.927625 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:35.927526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"]
Sep 06 05:03:35.930528 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:03:35.930498    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3c2667_c372_4159_97e5_369c98275a03.slice/crio-a5fad608a7f5097804e9f1a83e7060b7dea4e2563f61aec8771cf49d1ccf7b45 WatchSource:0}: Error finding container a5fad608a7f5097804e9f1a83e7060b7dea4e2563f61aec8771cf49d1ccf7b45: Status 404 returned error can't find the container with id a5fad608a7f5097804e9f1a83e7060b7dea4e2563f61aec8771cf49d1ccf7b45
Sep 06 05:03:36.015579 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.015547    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"]
Sep 06 05:03:36.018496 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.018477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.020918 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.020841    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 06 05:03:36.020918 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.020853    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 06 05:03:36.020918 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.020896    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 06 05:03:36.021297 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.020924    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 06 05:03:36.021297 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.020972    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-grg8c\""
Sep 06 05:03:36.030735 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.030044    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"]
Sep 06 05:03:36.143469 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.143433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqzt\" (UniqueName: \"kubernetes.io/projected/1fa736a2-0e9f-4018-add2-449fdf2d48c8-kube-api-access-mgqzt\") pod \"kube-auth-proxy-6c96787df8-bmmvf\" (UID: \"1fa736a2-0e9f-4018-add2-449fdf2d48c8\") " pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.143644 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.143479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fa736a2-0e9f-4018-add2-449fdf2d48c8-tmp\") pod \"kube-auth-proxy-6c96787df8-bmmvf\" (UID: \"1fa736a2-0e9f-4018-add2-449fdf2d48c8\") " pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.143688 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.143633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa736a2-0e9f-4018-add2-449fdf2d48c8-tls-certs\") pod \"kube-auth-proxy-6c96787df8-bmmvf\" (UID: \"1fa736a2-0e9f-4018-add2-449fdf2d48c8\") " pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.197608 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.197520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd" event={"ID":"3f3c2667-c372-4159-97e5-369c98275a03","Type":"ContainerStarted","Data":"a5fad608a7f5097804e9f1a83e7060b7dea4e2563f61aec8771cf49d1ccf7b45"}
Sep 06 05:03:36.245019 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.244987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa736a2-0e9f-4018-add2-449fdf2d48c8-tls-certs\") pod \"kube-auth-proxy-6c96787df8-bmmvf\" (UID: \"1fa736a2-0e9f-4018-add2-449fdf2d48c8\") " pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.245208 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.245027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqzt\" (UniqueName: \"kubernetes.io/projected/1fa736a2-0e9f-4018-add2-449fdf2d48c8-kube-api-access-mgqzt\") pod \"kube-auth-proxy-6c96787df8-bmmvf\" (UID: \"1fa736a2-0e9f-4018-add2-449fdf2d48c8\") " pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.245208 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.245049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fa736a2-0e9f-4018-add2-449fdf2d48c8-tmp\") pod \"kube-auth-proxy-6c96787df8-bmmvf\" (UID: \"1fa736a2-0e9f-4018-add2-449fdf2d48c8\") " pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.247504 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.247484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fa736a2-0e9f-4018-add2-449fdf2d48c8-tmp\") pod \"kube-auth-proxy-6c96787df8-bmmvf\" (UID: \"1fa736a2-0e9f-4018-add2-449fdf2d48c8\") " pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.247628 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.247611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa736a2-0e9f-4018-add2-449fdf2d48c8-tls-certs\") pod \"kube-auth-proxy-6c96787df8-bmmvf\" (UID: \"1fa736a2-0e9f-4018-add2-449fdf2d48c8\") " pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.252790 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.252770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqzt\" (UniqueName: \"kubernetes.io/projected/1fa736a2-0e9f-4018-add2-449fdf2d48c8-kube-api-access-mgqzt\") pod \"kube-auth-proxy-6c96787df8-bmmvf\" (UID: \"1fa736a2-0e9f-4018-add2-449fdf2d48c8\") " pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.332816 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.332789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"
Sep 06 05:03:36.620746 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:36.620701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf"]
Sep 06 05:03:36.622297 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:03:36.622268    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa736a2_0e9f_4018_add2_449fdf2d48c8.slice/crio-712363bdc3ebebedfea2336e3c9c7facd9bc25632c86c74e7e18f3bc7d242949 WatchSource:0}: Error finding container 712363bdc3ebebedfea2336e3c9c7facd9bc25632c86c74e7e18f3bc7d242949: Status 404 returned error can't find the container with id 712363bdc3ebebedfea2336e3c9c7facd9bc25632c86c74e7e18f3bc7d242949
Sep 06 05:03:37.205660 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:37.205624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf" event={"ID":"1fa736a2-0e9f-4018-add2-449fdf2d48c8","Type":"ContainerStarted","Data":"712363bdc3ebebedfea2336e3c9c7facd9bc25632c86c74e7e18f3bc7d242949"}
Sep 06 05:03:38.215018 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:38.214345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd" event={"ID":"3f3c2667-c372-4159-97e5-369c98275a03","Type":"ContainerStarted","Data":"f9204c20ef550826bba08dd8eae42f670637b4938143735956e9479c9d39dae0"}
Sep 06 05:03:38.215018 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:38.214471    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:03:38.230949 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:38.230891    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd" podStartSLOduration=1.86134756 podStartE2EDuration="3.230876351s" totalImagesPullingTime="1.369528791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:03:38.229326903 +0000 UTC m=+494.370886867" watchObservedRunningTime="2026-09-06 05:03:38.230876351 +0000 UTC m=+494.372436293"
Sep 06 05:03:40.228520 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:40.228489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf" event={"ID":"1fa736a2-0e9f-4018-add2-449fdf2d48c8","Type":"ContainerStarted","Data":"da24668b0279cb036d68b99a835782403108bb44b516886012adb788d121922b"}
Sep 06 05:03:40.243916 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:40.243851    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c96787df8-bmmvf" podStartSLOduration=2.353366023 podStartE2EDuration="5.243835724s" totalImagesPullingTime="2.890469701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:03:40.243097832 +0000 UTC m=+496.384657773" watchObservedRunningTime="2026-09-06 05:03:40.243835724 +0000 UTC m=+496.385395666"
Sep 06 05:03:49.224568 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:03:49.224538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7d8d45985c-gsjmd"
Sep 06 05:04:51.126475 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.126443    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"]
Sep 06 05:04:51.129936 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.129920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.132304 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.132278    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 06 05:04:51.132958 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.132939    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 06 05:04:51.132958 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.132941    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-k7tq2\""
Sep 06 05:04:51.136260 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.136237    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"]
Sep 06 05:04:51.222912 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.222846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.222912 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.222886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8jgx\" (UniqueName: \"kubernetes.io/projected/8e986b7d-4217-4d87-864b-61f8a96e43be-kube-api-access-b8jgx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.222912 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.222911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.225281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.225257    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"]
Sep 06 05:04:51.228672 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.228657    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.236037 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.236012    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"]
Sep 06 05:04:51.324074 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.324044    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"]
Sep 06 05:04:51.324343 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.324322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.324424 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.324367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.324424 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.324391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8jgx\" (UniqueName: \"kubernetes.io/projected/8e986b7d-4217-4d87-864b-61f8a96e43be-kube-api-access-b8jgx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.324516 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.324472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.324572 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.324520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lnb\" (UniqueName: \"kubernetes.io/projected/0477bb5a-cd99-4f99-ba97-8ed03a16d488-kube-api-access-g4lnb\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.324572 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.324563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.324772 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.324755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.324876 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.324830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.327698 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.327684    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.331614 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.331593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8jgx\" (UniqueName: \"kubernetes.io/projected/8e986b7d-4217-4d87-864b-61f8a96e43be-kube-api-access-b8jgx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.335314 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.335293    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"]
Sep 06 05:04:51.425569 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.425547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.425800 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.425781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.426017 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.425997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.426231 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.426011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.426231 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.426149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chj6q\" (UniqueName: \"kubernetes.io/projected/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-kube-api-access-chj6q\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.426385 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.426309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lnb\" (UniqueName: \"kubernetes.io/projected/0477bb5a-cd99-4f99-ba97-8ed03a16d488-kube-api-access-g4lnb\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.426385 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.426343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.426385 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.426367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.428436 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.428412    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"]
Sep 06 05:04:51.432600 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.432584    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.433620 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.433602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lnb\" (UniqueName: \"kubernetes.io/projected/0477bb5a-cd99-4f99-ba97-8ed03a16d488-kube-api-access-g4lnb\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.438763 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.438742    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"]
Sep 06 05:04:51.440808 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.440785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:04:51.527719 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.527688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.527834 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.527737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.527834 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.527781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.527834 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.527801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hmcm\" (UniqueName: \"kubernetes.io/projected/7ae9e8f7-6015-4c9e-8af2-326679600476-kube-api-access-8hmcm\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.527834 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.527827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.527992 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.527934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chj6q\" (UniqueName: \"kubernetes.io/projected/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-kube-api-access-chj6q\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.528194 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.528140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.528239 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.528176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.536404 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.536382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chj6q\" (UniqueName: \"kubernetes.io/projected/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-kube-api-access-chj6q\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.538187 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.538144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:51.628979 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.628952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.629091 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.629034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.629091 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.629078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hmcm\" (UniqueName: \"kubernetes.io/projected/7ae9e8f7-6015-4c9e-8af2-326679600476-kube-api-access-8hmcm\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.629390 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.629371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.629436 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.629422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.637090 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.637067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hmcm\" (UniqueName: \"kubernetes.io/projected/7ae9e8f7-6015-4c9e-8af2-326679600476-kube-api-access-8hmcm\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.648914 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.648899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:51.715746 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.715719    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"]
Sep 06 05:04:51.717927 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:04:51.717838    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e986b7d_4217_4d87_864b_61f8a96e43be.slice/crio-8b906f6565f5a018d8c23314788e24bc5afd45e30d3ddccf826ddeef1c414566 WatchSource:0}: Error finding container 8b906f6565f5a018d8c23314788e24bc5afd45e30d3ddccf826ddeef1c414566: Status 404 returned error can't find the container with id 8b906f6565f5a018d8c23314788e24bc5afd45e30d3ddccf826ddeef1c414566
Sep 06 05:04:51.768996 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.768975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:51.811436 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.811394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"]
Sep 06 05:04:51.813376 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:04:51.813349    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0477bb5a_cd99_4f99_ba97_8ed03a16d488.slice/crio-f4216f34103530b90ebd5b84db400cdf698f67925cd2d1906a7c5e0d4af0a722 WatchSource:0}: Error finding container f4216f34103530b90ebd5b84db400cdf698f67925cd2d1906a7c5e0d4af0a722: Status 404 returned error can't find the container with id f4216f34103530b90ebd5b84db400cdf698f67925cd2d1906a7c5e0d4af0a722
Sep 06 05:04:51.919015 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:51.918987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"]
Sep 06 05:04:51.922248 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:04:51.922221    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a97d31e_4bcc_4beb_b7fe_b0cfb6257e37.slice/crio-9141ef22c7d992197e58ee4696cd98d5cdecde8debfec609ac3e856b4805ec37 WatchSource:0}: Error finding container 9141ef22c7d992197e58ee4696cd98d5cdecde8debfec609ac3e856b4805ec37: Status 404 returned error can't find the container with id 9141ef22c7d992197e58ee4696cd98d5cdecde8debfec609ac3e856b4805ec37
Sep 06 05:04:52.043704 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.043681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"]
Sep 06 05:04:52.084045 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:04:52.084020    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae9e8f7_6015_4c9e_8af2_326679600476.slice/crio-4914a232ae6ab567e3eead615ff5c34d37cf76e267ff69409ed572c0da6a9b00 WatchSource:0}: Error finding container 4914a232ae6ab567e3eead615ff5c34d37cf76e267ff69409ed572c0da6a9b00: Status 404 returned error can't find the container with id 4914a232ae6ab567e3eead615ff5c34d37cf76e267ff69409ed572c0da6a9b00
Sep 06 05:04:52.577958 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.577923    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ae9e8f7-6015-4c9e-8af2-326679600476" containerID="e0656ddf9af9b7ee746aacfd4af1e3837560d0d55eee280441a47f02ad34786a" exitCode=0
Sep 06 05:04:52.578395 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.578011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq" event={"ID":"7ae9e8f7-6015-4c9e-8af2-326679600476","Type":"ContainerDied","Data":"e0656ddf9af9b7ee746aacfd4af1e3837560d0d55eee280441a47f02ad34786a"}
Sep 06 05:04:52.578395 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.578076    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq" event={"ID":"7ae9e8f7-6015-4c9e-8af2-326679600476","Type":"ContainerStarted","Data":"4914a232ae6ab567e3eead615ff5c34d37cf76e267ff69409ed572c0da6a9b00"}
Sep 06 05:04:52.579641 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.579621    2580 generic.go:356] "Generic (PLEG): container finished" podID="0477bb5a-cd99-4f99-ba97-8ed03a16d488" containerID="ff735ce67ede4820d00fa032083c6ebc8a3dad3e4829f8e5b626f30f5cb851a2" exitCode=0
Sep 06 05:04:52.579728 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.579684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl" event={"ID":"0477bb5a-cd99-4f99-ba97-8ed03a16d488","Type":"ContainerDied","Data":"ff735ce67ede4820d00fa032083c6ebc8a3dad3e4829f8e5b626f30f5cb851a2"}
Sep 06 05:04:52.579782 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.579729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl" event={"ID":"0477bb5a-cd99-4f99-ba97-8ed03a16d488","Type":"ContainerStarted","Data":"f4216f34103530b90ebd5b84db400cdf698f67925cd2d1906a7c5e0d4af0a722"}
Sep 06 05:04:52.581410 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.581388    2580 generic.go:356] "Generic (PLEG): container finished" podID="7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" containerID="3c39d5bc3f1150879874e58f737f77ba60254e74b3d54bc837c64c0d078a71c9" exitCode=0
Sep 06 05:04:52.581517 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.581418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98" event={"ID":"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37","Type":"ContainerDied","Data":"3c39d5bc3f1150879874e58f737f77ba60254e74b3d54bc837c64c0d078a71c9"}
Sep 06 05:04:52.581517 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.581475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98" event={"ID":"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37","Type":"ContainerStarted","Data":"9141ef22c7d992197e58ee4696cd98d5cdecde8debfec609ac3e856b4805ec37"}
Sep 06 05:04:52.583701 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.583683    2580 generic.go:356] "Generic (PLEG): container finished" podID="8e986b7d-4217-4d87-864b-61f8a96e43be" containerID="3094fb0ab66b9e2022acae4bd35e1c276830c4da276076553d5b04de75ab516a" exitCode=0
Sep 06 05:04:52.583792 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.583741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7" event={"ID":"8e986b7d-4217-4d87-864b-61f8a96e43be","Type":"ContainerDied","Data":"3094fb0ab66b9e2022acae4bd35e1c276830c4da276076553d5b04de75ab516a"}
Sep 06 05:04:52.583792 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:52.583764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7" event={"ID":"8e986b7d-4217-4d87-864b-61f8a96e43be","Type":"ContainerStarted","Data":"8b906f6565f5a018d8c23314788e24bc5afd45e30d3ddccf826ddeef1c414566"}
Sep 06 05:04:54.597249 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:54.597217    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ae9e8f7-6015-4c9e-8af2-326679600476" containerID="49d37b80b33cfaa4bcb6d67d7a732307a9d65a7140d0251ebd1b8abb3dc67201" exitCode=0
Sep 06 05:04:54.597550 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:54.597303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq" event={"ID":"7ae9e8f7-6015-4c9e-8af2-326679600476","Type":"ContainerDied","Data":"49d37b80b33cfaa4bcb6d67d7a732307a9d65a7140d0251ebd1b8abb3dc67201"}
Sep 06 05:04:54.599342 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:54.599298    2580 generic.go:356] "Generic (PLEG): container finished" podID="7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" containerID="55f8864ae0087ddbc6ef2c8175917e667611e8bcd1351a7700b2dcbcc83838fd" exitCode=0
Sep 06 05:04:54.599342 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:54.599331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98" event={"ID":"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37","Type":"ContainerDied","Data":"55f8864ae0087ddbc6ef2c8175917e667611e8bcd1351a7700b2dcbcc83838fd"}
Sep 06 05:04:55.606514 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:55.606474    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ae9e8f7-6015-4c9e-8af2-326679600476" containerID="83bdafff1edc0b8bfac1d2c10a6373fb637a0f3b0332d6bcf8d473f905b79e0b" exitCode=0
Sep 06 05:04:55.606888 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:55.606556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq" event={"ID":"7ae9e8f7-6015-4c9e-8af2-326679600476","Type":"ContainerDied","Data":"83bdafff1edc0b8bfac1d2c10a6373fb637a0f3b0332d6bcf8d473f905b79e0b"}
Sep 06 05:04:55.608866 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:55.608843    2580 generic.go:356] "Generic (PLEG): container finished" podID="7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" containerID="35f9f322ae26aec83231ac7c9ae0db748196bf4fe24a83d548514b96c2d51b13" exitCode=0
Sep 06 05:04:55.608977 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:55.608875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98" event={"ID":"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37","Type":"ContainerDied","Data":"35f9f322ae26aec83231ac7c9ae0db748196bf4fe24a83d548514b96c2d51b13"}
Sep 06 05:04:56.757135 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.757114    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:56.775568 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.775545    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:56.872010 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.871937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-bundle\") pod \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") "
Sep 06 05:04:56.872010 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.871978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chj6q\" (UniqueName: \"kubernetes.io/projected/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-kube-api-access-chj6q\") pod \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") "
Sep 06 05:04:56.872237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.872043    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-bundle\") pod \"7ae9e8f7-6015-4c9e-8af2-326679600476\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") "
Sep 06 05:04:56.872237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.872086    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-util\") pod \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\" (UID: \"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37\") "
Sep 06 05:04:56.872340 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.872237    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hmcm\" (UniqueName: \"kubernetes.io/projected/7ae9e8f7-6015-4c9e-8af2-326679600476-kube-api-access-8hmcm\") pod \"7ae9e8f7-6015-4c9e-8af2-326679600476\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") "
Sep 06 05:04:56.872393 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.872352    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-util\") pod \"7ae9e8f7-6015-4c9e-8af2-326679600476\" (UID: \"7ae9e8f7-6015-4c9e-8af2-326679600476\") "
Sep 06 05:04:56.872702 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.872519    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-bundle" (OuterVolumeSpecName: "bundle") pod "7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" (UID: "7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:04:56.872702 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.872653    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-bundle\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:04:56.872860 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.872802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-bundle" (OuterVolumeSpecName: "bundle") pod "7ae9e8f7-6015-4c9e-8af2-326679600476" (UID: "7ae9e8f7-6015-4c9e-8af2-326679600476"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:04:56.874452 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.874429    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-kube-api-access-chj6q" (OuterVolumeSpecName: "kube-api-access-chj6q") pod "7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" (UID: "7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37"). InnerVolumeSpecName "kube-api-access-chj6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:04:56.874748 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.874728    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae9e8f7-6015-4c9e-8af2-326679600476-kube-api-access-8hmcm" (OuterVolumeSpecName: "kube-api-access-8hmcm") pod "7ae9e8f7-6015-4c9e-8af2-326679600476" (UID: "7ae9e8f7-6015-4c9e-8af2-326679600476"). InnerVolumeSpecName "kube-api-access-8hmcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:04:56.881019 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.880992    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-util" (OuterVolumeSpecName: "util") pod "7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" (UID: "7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:04:56.881438 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.881418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-util" (OuterVolumeSpecName: "util") pod "7ae9e8f7-6015-4c9e-8af2-326679600476" (UID: "7ae9e8f7-6015-4c9e-8af2-326679600476"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:04:56.973369 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.973342    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chj6q\" (UniqueName: \"kubernetes.io/projected/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-kube-api-access-chj6q\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:04:56.973369 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.973368    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-bundle\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:04:56.973515 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.973381    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37-util\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:04:56.973515 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.973392    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hmcm\" (UniqueName: \"kubernetes.io/projected/7ae9e8f7-6015-4c9e-8af2-326679600476-kube-api-access-8hmcm\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:04:56.973515 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:56.973404    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ae9e8f7-6015-4c9e-8af2-326679600476-util\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:04:57.622188 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:57.622150    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98"
Sep 06 05:04:57.622338 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:57.622185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbf6k98" event={"ID":"7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37","Type":"ContainerDied","Data":"9141ef22c7d992197e58ee4696cd98d5cdecde8debfec609ac3e856b4805ec37"}
Sep 06 05:04:57.622338 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:57.622239    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9141ef22c7d992197e58ee4696cd98d5cdecde8debfec609ac3e856b4805ec37"
Sep 06 05:04:57.624232 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:57.624217    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq"
Sep 06 05:04:57.624319 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:57.624256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053tdxlq" event={"ID":"7ae9e8f7-6015-4c9e-8af2-326679600476","Type":"ContainerDied","Data":"4914a232ae6ab567e3eead615ff5c34d37cf76e267ff69409ed572c0da6a9b00"}
Sep 06 05:04:57.624319 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:57.624295    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4914a232ae6ab567e3eead615ff5c34d37cf76e267ff69409ed572c0da6a9b00"
Sep 06 05:04:57.626367 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:57.626342    2580 generic.go:356] "Generic (PLEG): container finished" podID="0477bb5a-cd99-4f99-ba97-8ed03a16d488" containerID="003c5989cd2bca1662105ee28355911418bd6b4c90154dc204b96c2e193b0d34" exitCode=0
Sep 06 05:04:57.626459 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:57.626390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl" event={"ID":"0477bb5a-cd99-4f99-ba97-8ed03a16d488","Type":"ContainerDied","Data":"003c5989cd2bca1662105ee28355911418bd6b4c90154dc204b96c2e193b0d34"}
Sep 06 05:04:58.633001 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:58.632968    2580 generic.go:356] "Generic (PLEG): container finished" podID="0477bb5a-cd99-4f99-ba97-8ed03a16d488" containerID="b502a9bb91bd8ce6433d5e3cc7f569c83c18a0c4fdc27c98405071a3b317e7ef" exitCode=0
Sep 06 05:04:58.633405 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:58.633056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl" event={"ID":"0477bb5a-cd99-4f99-ba97-8ed03a16d488","Type":"ContainerDied","Data":"b502a9bb91bd8ce6433d5e3cc7f569c83c18a0c4fdc27c98405071a3b317e7ef"}
Sep 06 05:04:59.763059 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:59.763038    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:04:59.894967 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:59.894903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-util\") pod \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") "
Sep 06 05:04:59.894967 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:59.894933    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4lnb\" (UniqueName: \"kubernetes.io/projected/0477bb5a-cd99-4f99-ba97-8ed03a16d488-kube-api-access-g4lnb\") pod \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") "
Sep 06 05:04:59.895128 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:59.894995    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-bundle\") pod \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\" (UID: \"0477bb5a-cd99-4f99-ba97-8ed03a16d488\") "
Sep 06 05:04:59.895584 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:59.895549    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-bundle" (OuterVolumeSpecName: "bundle") pod "0477bb5a-cd99-4f99-ba97-8ed03a16d488" (UID: "0477bb5a-cd99-4f99-ba97-8ed03a16d488"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:04:59.897201 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:59.897182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0477bb5a-cd99-4f99-ba97-8ed03a16d488-kube-api-access-g4lnb" (OuterVolumeSpecName: "kube-api-access-g4lnb") pod "0477bb5a-cd99-4f99-ba97-8ed03a16d488" (UID: "0477bb5a-cd99-4f99-ba97-8ed03a16d488"). InnerVolumeSpecName "kube-api-access-g4lnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:04:59.899342 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:59.899322    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-util" (OuterVolumeSpecName: "util") pod "0477bb5a-cd99-4f99-ba97-8ed03a16d488" (UID: "0477bb5a-cd99-4f99-ba97-8ed03a16d488"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:04:59.995860 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:59.995836    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-util\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:04:59.995860 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:59.995857    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4lnb\" (UniqueName: \"kubernetes.io/projected/0477bb5a-cd99-4f99-ba97-8ed03a16d488-kube-api-access-g4lnb\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:04:59.995992 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:04:59.995867    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0477bb5a-cd99-4f99-ba97-8ed03a16d488-bundle\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:05:00.646069 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:00.645993    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl"
Sep 06 05:05:00.646259 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:00.645988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9f8kl" event={"ID":"0477bb5a-cd99-4f99-ba97-8ed03a16d488","Type":"ContainerDied","Data":"f4216f34103530b90ebd5b84db400cdf698f67925cd2d1906a7c5e0d4af0a722"}
Sep 06 05:05:00.646259 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:00.646110    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4216f34103530b90ebd5b84db400cdf698f67925cd2d1906a7c5e0d4af0a722"
Sep 06 05:05:15.724170 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:15.724120    2580 generic.go:356] "Generic (PLEG): container finished" podID="8e986b7d-4217-4d87-864b-61f8a96e43be" containerID="98b00bc1b1fb48463c4244360d4de47d0fc7eb81b20e57584cad1522fcb5d270" exitCode=0
Sep 06 05:05:15.724557 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:15.724195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7" event={"ID":"8e986b7d-4217-4d87-864b-61f8a96e43be","Type":"ContainerDied","Data":"98b00bc1b1fb48463c4244360d4de47d0fc7eb81b20e57584cad1522fcb5d270"}
Sep 06 05:05:16.731862 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:16.731825    2580 generic.go:356] "Generic (PLEG): container finished" podID="8e986b7d-4217-4d87-864b-61f8a96e43be" containerID="37c605789bd90299b556043d4efe2aadfb2427b15f931b7260ac7a620616bba4" exitCode=0
Sep 06 05:05:16.732259 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:16.731866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7" event={"ID":"8e986b7d-4217-4d87-864b-61f8a96e43be","Type":"ContainerDied","Data":"37c605789bd90299b556043d4efe2aadfb2427b15f931b7260ac7a620616bba4"}
Sep 06 05:05:17.859634 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:17.859603    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:05:18.032780 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.032711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-util\") pod \"8e986b7d-4217-4d87-864b-61f8a96e43be\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") "
Sep 06 05:05:18.032780 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.032751    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8jgx\" (UniqueName: \"kubernetes.io/projected/8e986b7d-4217-4d87-864b-61f8a96e43be-kube-api-access-b8jgx\") pod \"8e986b7d-4217-4d87-864b-61f8a96e43be\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") "
Sep 06 05:05:18.032941 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.032818    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-bundle\") pod \"8e986b7d-4217-4d87-864b-61f8a96e43be\" (UID: \"8e986b7d-4217-4d87-864b-61f8a96e43be\") "
Sep 06 05:05:18.033345 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.033314    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-bundle" (OuterVolumeSpecName: "bundle") pod "8e986b7d-4217-4d87-864b-61f8a96e43be" (UID: "8e986b7d-4217-4d87-864b-61f8a96e43be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:05:18.034909 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.034877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e986b7d-4217-4d87-864b-61f8a96e43be-kube-api-access-b8jgx" (OuterVolumeSpecName: "kube-api-access-b8jgx") pod "8e986b7d-4217-4d87-864b-61f8a96e43be" (UID: "8e986b7d-4217-4d87-864b-61f8a96e43be"). InnerVolumeSpecName "kube-api-access-b8jgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:05:18.039231 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.039202    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-util" (OuterVolumeSpecName: "util") pod "8e986b7d-4217-4d87-864b-61f8a96e43be" (UID: "8e986b7d-4217-4d87-864b-61f8a96e43be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:05:18.134120 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.134092    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-bundle\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:05:18.134120 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.134115    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e986b7d-4217-4d87-864b-61f8a96e43be-util\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:05:18.134283 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.134124    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8jgx\" (UniqueName: \"kubernetes.io/projected/8e986b7d-4217-4d87-864b-61f8a96e43be-kube-api-access-b8jgx\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:05:18.744504 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.744472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7" event={"ID":"8e986b7d-4217-4d87-864b-61f8a96e43be","Type":"ContainerDied","Data":"8b906f6565f5a018d8c23314788e24bc5afd45e30d3ddccf826ddeef1c414566"}
Sep 06 05:05:18.744665 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.744526    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b906f6565f5a018d8c23314788e24bc5afd45e30d3ddccf826ddeef1c414566"
Sep 06 05:05:18.744665 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:18.744486    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0xcgt7"
Sep 06 05:05:24.354340 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:24.354314    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:05:24.354765 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:24.354349    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:05:33.779053 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779021    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"]
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779352    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" containerName="pull"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779364    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" containerName="pull"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779374    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae9e8f7-6015-4c9e-8af2-326679600476" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779379    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae9e8f7-6015-4c9e-8af2-326679600476" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779385    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779391    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779399    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" containerName="util"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779403    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" containerName="util"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779418    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0477bb5a-cd99-4f99-ba97-8ed03a16d488" containerName="pull"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779424    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0477bb5a-cd99-4f99-ba97-8ed03a16d488" containerName="pull"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779430    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0477bb5a-cd99-4f99-ba97-8ed03a16d488" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779435    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0477bb5a-cd99-4f99-ba97-8ed03a16d488" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779446    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae9e8f7-6015-4c9e-8af2-326679600476" containerName="util"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779451    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae9e8f7-6015-4c9e-8af2-326679600476" containerName="util"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779457    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e986b7d-4217-4d87-864b-61f8a96e43be" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779462    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e986b7d-4217-4d87-864b-61f8a96e43be" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779469    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0477bb5a-cd99-4f99-ba97-8ed03a16d488" containerName="util"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779474    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0477bb5a-cd99-4f99-ba97-8ed03a16d488" containerName="util"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779482    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e986b7d-4217-4d87-864b-61f8a96e43be" containerName="util"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779487    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e986b7d-4217-4d87-864b-61f8a96e43be" containerName="util"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779492    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae9e8f7-6015-4c9e-8af2-326679600476" containerName="pull"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779497    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae9e8f7-6015-4c9e-8af2-326679600476" containerName="pull"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779506    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e986b7d-4217-4d87-864b-61f8a96e43be" containerName="pull"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779510    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e986b7d-4217-4d87-864b-61f8a96e43be" containerName="pull"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779561    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e986b7d-4217-4d87-864b-61f8a96e43be" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779569    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a97d31e-4bcc-4beb-b7fe-b0cfb6257e37" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779573    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae9e8f7-6015-4c9e-8af2-326679600476" containerName="extract"
Sep 06 05:05:33.781463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.779581    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0477bb5a-cd99-4f99-ba97-8ed03a16d488" containerName="extract"
Sep 06 05:05:33.782479 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.782464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:33.784956 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.784930    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 06 05:05:33.784956 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.784932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 06 05:05:33.785782 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.785765    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2tg77\""
Sep 06 05:05:33.785851 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.785782    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 06 05:05:33.785851 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.785774    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 06 05:05:33.791829 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.791810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"]
Sep 06 05:05:33.848251 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.848219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c3b7076-9e38-4518-88aa-4e8f62b1554a-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rjg5t\" (UID: \"7c3b7076-9e38-4518-88aa-4e8f62b1554a\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:33.848251 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.848253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvdbc\" (UniqueName: \"kubernetes.io/projected/7c3b7076-9e38-4518-88aa-4e8f62b1554a-kube-api-access-bvdbc\") pod \"kuadrant-console-plugin-598f86468b-rjg5t\" (UID: \"7c3b7076-9e38-4518-88aa-4e8f62b1554a\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:33.848422 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.848276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c3b7076-9e38-4518-88aa-4e8f62b1554a-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rjg5t\" (UID: \"7c3b7076-9e38-4518-88aa-4e8f62b1554a\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:33.948691 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.948663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c3b7076-9e38-4518-88aa-4e8f62b1554a-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rjg5t\" (UID: \"7c3b7076-9e38-4518-88aa-4e8f62b1554a\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:33.948799 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.948749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c3b7076-9e38-4518-88aa-4e8f62b1554a-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rjg5t\" (UID: \"7c3b7076-9e38-4518-88aa-4e8f62b1554a\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:33.948799 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.948778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvdbc\" (UniqueName: \"kubernetes.io/projected/7c3b7076-9e38-4518-88aa-4e8f62b1554a-kube-api-access-bvdbc\") pod \"kuadrant-console-plugin-598f86468b-rjg5t\" (UID: \"7c3b7076-9e38-4518-88aa-4e8f62b1554a\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:33.948883 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:05:33.948806    2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 06 05:05:33.948883 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:05:33.948870    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3b7076-9e38-4518-88aa-4e8f62b1554a-plugin-serving-cert podName:7c3b7076-9e38-4518-88aa-4e8f62b1554a nodeName:}" failed. No retries permitted until 2026-09-06 05:05:34.448850886 +0000 UTC m=+610.590410806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7c3b7076-9e38-4518-88aa-4e8f62b1554a-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-rjg5t" (UID: "7c3b7076-9e38-4518-88aa-4e8f62b1554a") : secret "plugin-serving-cert" not found
Sep 06 05:05:33.949519 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.949495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c3b7076-9e38-4518-88aa-4e8f62b1554a-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rjg5t\" (UID: \"7c3b7076-9e38-4518-88aa-4e8f62b1554a\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:33.957569 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:33.957548    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvdbc\" (UniqueName: \"kubernetes.io/projected/7c3b7076-9e38-4518-88aa-4e8f62b1554a-kube-api-access-bvdbc\") pod \"kuadrant-console-plugin-598f86468b-rjg5t\" (UID: \"7c3b7076-9e38-4518-88aa-4e8f62b1554a\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:34.453991 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:34.453511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c3b7076-9e38-4518-88aa-4e8f62b1554a-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rjg5t\" (UID: \"7c3b7076-9e38-4518-88aa-4e8f62b1554a\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:34.456975 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:34.456942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c3b7076-9e38-4518-88aa-4e8f62b1554a-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rjg5t\" (UID: \"7c3b7076-9e38-4518-88aa-4e8f62b1554a\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:34.693398 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:34.693361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"
Sep 06 05:05:34.964353 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:34.964328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t"]
Sep 06 05:05:34.965824 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:05:34.965800    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c3b7076_9e38_4518_88aa_4e8f62b1554a.slice/crio-1cdbd4631f3f07213f8ce24b7faa0a1feac5ea1565c8b9253bdabe662b2dac07 WatchSource:0}: Error finding container 1cdbd4631f3f07213f8ce24b7faa0a1feac5ea1565c8b9253bdabe662b2dac07: Status 404 returned error can't find the container with id 1cdbd4631f3f07213f8ce24b7faa0a1feac5ea1565c8b9253bdabe662b2dac07
Sep 06 05:05:35.829658 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:35.829623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t" event={"ID":"7c3b7076-9e38-4518-88aa-4e8f62b1554a","Type":"ContainerStarted","Data":"1cdbd4631f3f07213f8ce24b7faa0a1feac5ea1565c8b9253bdabe662b2dac07"}
Sep 06 05:05:42.872132 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:42.872098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t" event={"ID":"7c3b7076-9e38-4518-88aa-4e8f62b1554a","Type":"ContainerStarted","Data":"3dc96e9ca5024780ae8cb312aad85b09ddc3cf4c2a5e1b205297378c9d96e3d9"}
Sep 06 05:05:42.891540 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:05:42.891492    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rjg5t" podStartSLOduration=2.497986595 podStartE2EDuration="9.8914792s" totalImagesPullingTime="7.393492605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:05:42.89060728 +0000 UTC m=+619.032167225" watchObservedRunningTime="2026-09-06 05:05:42.8914792 +0000 UTC m=+619.033039142"
Sep 06 05:06:15.343401 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.343363    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:06:15.361406 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.361379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:06:15.361565 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.361467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"
Sep 06 05:06:15.363746 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.363725    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 06 05:06:15.370563 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.370532    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:06:15.471845 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.471821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhk9c\" (UniqueName: \"kubernetes.io/projected/29570032-d53c-48cc-95b0-857a2933b80b-kube-api-access-jhk9c\") pod \"limitador-limitador-85d9db4d5d-vr8tg\" (UID: \"29570032-d53c-48cc-95b0-857a2933b80b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"
Sep 06 05:06:15.471961 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.471894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/29570032-d53c-48cc-95b0-857a2933b80b-config-file\") pod \"limitador-limitador-85d9db4d5d-vr8tg\" (UID: \"29570032-d53c-48cc-95b0-857a2933b80b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"
Sep 06 05:06:15.572489 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.572456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/29570032-d53c-48cc-95b0-857a2933b80b-config-file\") pod \"limitador-limitador-85d9db4d5d-vr8tg\" (UID: \"29570032-d53c-48cc-95b0-857a2933b80b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"
Sep 06 05:06:15.572682 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.572506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhk9c\" (UniqueName: \"kubernetes.io/projected/29570032-d53c-48cc-95b0-857a2933b80b-kube-api-access-jhk9c\") pod \"limitador-limitador-85d9db4d5d-vr8tg\" (UID: \"29570032-d53c-48cc-95b0-857a2933b80b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"
Sep 06 05:06:15.573109 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.573084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/29570032-d53c-48cc-95b0-857a2933b80b-config-file\") pod \"limitador-limitador-85d9db4d5d-vr8tg\" (UID: \"29570032-d53c-48cc-95b0-857a2933b80b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"
Sep 06 05:06:15.579801 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.579781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhk9c\" (UniqueName: \"kubernetes.io/projected/29570032-d53c-48cc-95b0-857a2933b80b-kube-api-access-jhk9c\") pod \"limitador-limitador-85d9db4d5d-vr8tg\" (UID: \"29570032-d53c-48cc-95b0-857a2933b80b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"
Sep 06 05:06:15.672012 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.671951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"
Sep 06 05:06:15.862175 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.862134    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6rbxp"]
Sep 06 05:06:15.865288 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.865274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6rbxp"
Sep 06 05:06:15.867505 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.867485    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-phjph\""
Sep 06 05:06:15.871558 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.871534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6rbxp"]
Sep 06 05:06:15.942328 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.942306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:06:15.944692 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:06:15.944655    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29570032_d53c_48cc_95b0_857a2933b80b.slice/crio-98c32588bbf49239a6601ee9a5505dfb43fa3f0cc0bc4b2cd68938c13eaf9d1a WatchSource:0}: Error finding container 98c32588bbf49239a6601ee9a5505dfb43fa3f0cc0bc4b2cd68938c13eaf9d1a: Status 404 returned error can't find the container with id 98c32588bbf49239a6601ee9a5505dfb43fa3f0cc0bc4b2cd68938c13eaf9d1a
Sep 06 05:06:15.946662 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.946639    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 05:06:15.976521 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:15.976496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69fvg\" (UniqueName: \"kubernetes.io/projected/81e8bb2f-dd89-43f0-b0d7-92e748248a12-kube-api-access-69fvg\") pod \"authorino-c548b5d7d-6rbxp\" (UID: \"81e8bb2f-dd89-43f0-b0d7-92e748248a12\") " pod="kuadrant-system/authorino-c548b5d7d-6rbxp"
Sep 06 05:06:16.035670 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:16.035644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg" event={"ID":"29570032-d53c-48cc-95b0-857a2933b80b","Type":"ContainerStarted","Data":"98c32588bbf49239a6601ee9a5505dfb43fa3f0cc0bc4b2cd68938c13eaf9d1a"}
Sep 06 05:06:16.077397 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:16.077368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69fvg\" (UniqueName: \"kubernetes.io/projected/81e8bb2f-dd89-43f0-b0d7-92e748248a12-kube-api-access-69fvg\") pod \"authorino-c548b5d7d-6rbxp\" (UID: \"81e8bb2f-dd89-43f0-b0d7-92e748248a12\") " pod="kuadrant-system/authorino-c548b5d7d-6rbxp"
Sep 06 05:06:16.084796 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:16.084769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69fvg\" (UniqueName: \"kubernetes.io/projected/81e8bb2f-dd89-43f0-b0d7-92e748248a12-kube-api-access-69fvg\") pod \"authorino-c548b5d7d-6rbxp\" (UID: \"81e8bb2f-dd89-43f0-b0d7-92e748248a12\") " pod="kuadrant-system/authorino-c548b5d7d-6rbxp"
Sep 06 05:06:16.176433 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:16.176408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6rbxp"
Sep 06 05:06:16.448829 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:16.448803    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6rbxp"]
Sep 06 05:06:16.450661 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:06:16.450638    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81e8bb2f_dd89_43f0_b0d7_92e748248a12.slice/crio-e9929afe78b324bf96838a3ddea42ea64c930faee9b0113fae949141281c8e08 WatchSource:0}: Error finding container e9929afe78b324bf96838a3ddea42ea64c930faee9b0113fae949141281c8e08: Status 404 returned error can't find the container with id e9929afe78b324bf96838a3ddea42ea64c930faee9b0113fae949141281c8e08
Sep 06 05:06:17.045192 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:17.045139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6rbxp" event={"ID":"81e8bb2f-dd89-43f0-b0d7-92e748248a12","Type":"ContainerStarted","Data":"e9929afe78b324bf96838a3ddea42ea64c930faee9b0113fae949141281c8e08"}
Sep 06 05:06:20.064562 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:20.064529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6rbxp" event={"ID":"81e8bb2f-dd89-43f0-b0d7-92e748248a12","Type":"ContainerStarted","Data":"3eb8a673c87f245f14e3c33739c65fd93df65f6c05e26f1c81d6cc8f7a4be8db"}
Sep 06 05:06:20.066379 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:20.066348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg" event={"ID":"29570032-d53c-48cc-95b0-857a2933b80b","Type":"ContainerStarted","Data":"8a237877eb234d221cb0fb50a377f433eb2d990e25e61f7507f9003b519dff88"}
Sep 06 05:06:20.066520 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:20.066407    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"
Sep 06 05:06:20.097315 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:20.097272    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-6rbxp" podStartSLOduration=2.479075386 podStartE2EDuration="5.097260632s" totalImagesPullingTime="2.618185246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:06:20.095740063 +0000 UTC m=+656.237300016" watchObservedRunningTime="2026-09-06 05:06:20.097260632 +0000 UTC m=+656.238820574"
Sep 06 05:06:20.140931 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:20.140875    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg" podStartSLOduration=2.021766366 podStartE2EDuration="5.140855606s" totalImagesPullingTime="3.11908924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:06:20.13720989 +0000 UTC m=+656.278769855" watchObservedRunningTime="2026-09-06 05:06:20.140855606 +0000 UTC m=+656.282415551"
Sep 06 05:06:31.072708 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:31.072676    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"
Sep 06 05:06:51.088947 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:51.088917    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6rbxp"]
Sep 06 05:06:51.089473 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:51.089088    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-6rbxp" podUID="81e8bb2f-dd89-43f0-b0d7-92e748248a12" containerName="authorino" containerID="cri-o://3eb8a673c87f245f14e3c33739c65fd93df65f6c05e26f1c81d6cc8f7a4be8db" gracePeriod=30
Sep 06 05:06:51.230330 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:51.230295    2580 generic.go:356] "Generic (PLEG): container finished" podID="81e8bb2f-dd89-43f0-b0d7-92e748248a12" containerID="3eb8a673c87f245f14e3c33739c65fd93df65f6c05e26f1c81d6cc8f7a4be8db" exitCode=0
Sep 06 05:06:51.230470 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:51.230365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6rbxp" event={"ID":"81e8bb2f-dd89-43f0-b0d7-92e748248a12","Type":"ContainerDied","Data":"3eb8a673c87f245f14e3c33739c65fd93df65f6c05e26f1c81d6cc8f7a4be8db"}
Sep 06 05:06:51.323577 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:51.323556    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6rbxp"
Sep 06 05:06:51.459939 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:51.459909    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69fvg\" (UniqueName: \"kubernetes.io/projected/81e8bb2f-dd89-43f0-b0d7-92e748248a12-kube-api-access-69fvg\") pod \"81e8bb2f-dd89-43f0-b0d7-92e748248a12\" (UID: \"81e8bb2f-dd89-43f0-b0d7-92e748248a12\") "
Sep 06 05:06:51.462194 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:51.462173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e8bb2f-dd89-43f0-b0d7-92e748248a12-kube-api-access-69fvg" (OuterVolumeSpecName: "kube-api-access-69fvg") pod "81e8bb2f-dd89-43f0-b0d7-92e748248a12" (UID: "81e8bb2f-dd89-43f0-b0d7-92e748248a12"). InnerVolumeSpecName "kube-api-access-69fvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:06:51.560789 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:51.560765    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69fvg\" (UniqueName: \"kubernetes.io/projected/81e8bb2f-dd89-43f0-b0d7-92e748248a12-kube-api-access-69fvg\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:06:52.237103 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:52.237069    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6rbxp"
Sep 06 05:06:52.237560 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:52.237071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6rbxp" event={"ID":"81e8bb2f-dd89-43f0-b0d7-92e748248a12","Type":"ContainerDied","Data":"e9929afe78b324bf96838a3ddea42ea64c930faee9b0113fae949141281c8e08"}
Sep 06 05:06:52.237560 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:52.237217    2580 scope.go:117] "RemoveContainer" containerID="3eb8a673c87f245f14e3c33739c65fd93df65f6c05e26f1c81d6cc8f7a4be8db"
Sep 06 05:06:52.256367 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:52.256340    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6rbxp"]
Sep 06 05:06:52.259757 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:52.259737    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6rbxp"]
Sep 06 05:06:52.346852 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:06:52.346829    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e8bb2f-dd89-43f0-b0d7-92e748248a12" path="/var/lib/kubelet/pods/81e8bb2f-dd89-43f0-b0d7-92e748248a12/volumes"
Sep 06 05:07:32.315189 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:07:32.315129    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:07:50.935469 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:07:50.935436    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:08:12.736979 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:12.736943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:08:21.543359 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.543325    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"]
Sep 06 05:08:21.543902 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.543885    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e8bb2f-dd89-43f0-b0d7-92e748248a12" containerName="authorino"
Sep 06 05:08:21.543951 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.543905    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e8bb2f-dd89-43f0-b0d7-92e748248a12" containerName="authorino"
Sep 06 05:08:21.544005 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.543994    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e8bb2f-dd89-43f0-b0d7-92e748248a12" containerName="authorino"
Sep 06 05:08:21.547069 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.547049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.549374 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.549328    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 06 05:08:21.549374 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.549356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 06 05:08:21.549374 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.549373    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9srnm\""
Sep 06 05:08:21.549597 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.549329    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 06 05:08:21.556117 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.556097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"]
Sep 06 05:08:21.564946 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.564924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.565046 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.564954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.565046 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.564986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq79p\" (UniqueName: \"kubernetes.io/projected/2fb801d7-cd7e-4371-b19f-57625e399e3e-kube-api-access-mq79p\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.565046 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.565027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.565179 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.565092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb801d7-cd7e-4371-b19f-57625e399e3e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.565179 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.565116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.666144 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.666119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.666282 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.666186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb801d7-cd7e-4371-b19f-57625e399e3e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.666282 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.666211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.666282 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.666256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.666440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.666281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.666440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.666334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq79p\" (UniqueName: \"kubernetes.io/projected/2fb801d7-cd7e-4371-b19f-57625e399e3e-kube-api-access-mq79p\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.666621 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.666579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.666621 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.666613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.666736 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.666650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.668552 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.668532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb801d7-cd7e-4371-b19f-57625e399e3e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.668873 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.668855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb801d7-cd7e-4371-b19f-57625e399e3e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.673613 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.673594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq79p\" (UniqueName: \"kubernetes.io/projected/2fb801d7-cd7e-4371-b19f-57625e399e3e-kube-api-access-mq79p\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq\" (UID: \"2fb801d7-cd7e-4371-b19f-57625e399e3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:21.858640 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:21.858569    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:22.129258 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:22.129231    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"]
Sep 06 05:08:22.130902 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:08:22.130876    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb801d7_cd7e_4371_b19f_57625e399e3e.slice/crio-ca017de88695f145649bf06c6c1bb2bb41281851e4c05de6c7938eeceeee36cf WatchSource:0}: Error finding container ca017de88695f145649bf06c6c1bb2bb41281851e4c05de6c7938eeceeee36cf: Status 404 returned error can't find the container with id ca017de88695f145649bf06c6c1bb2bb41281851e4c05de6c7938eeceeee36cf
Sep 06 05:08:22.353357 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:22.353293    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:08:22.706987 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:22.706953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq" event={"ID":"2fb801d7-cd7e-4371-b19f-57625e399e3e","Type":"ContainerStarted","Data":"ca017de88695f145649bf06c6c1bb2bb41281851e4c05de6c7938eeceeee36cf"}
Sep 06 05:08:23.713560 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:23.713529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq" event={"ID":"2fb801d7-cd7e-4371-b19f-57625e399e3e","Type":"ContainerStarted","Data":"86dc1f413cdc425b12e2f481bf5246ca053a9075ee3619ca90086e46529cbde6"}
Sep 06 05:08:23.730412 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:23.730361    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq" podStartSLOduration=1.746045533 podStartE2EDuration="2.730346123s" totalImagesPullingTime="984.30059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:08:23.728108955 +0000 UTC m=+779.869668895" watchObservedRunningTime="2026-09-06 05:08:23.730346123 +0000 UTC m=+779.871906055"
Sep 06 05:08:31.859666 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:31.859635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:31.860195 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:31.859713    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:31.872275 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:31.872255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:32.770899 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:32.770873    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq"
Sep 06 05:08:39.848232 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.848198    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"]
Sep 06 05:08:39.854304 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.854283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.856461 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.856441    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 06 05:08:39.861136 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.861114    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"]
Sep 06 05:08:39.895079 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.895059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.895222 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.895092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfwd\" (UniqueName: \"kubernetes.io/projected/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-kube-api-access-sqfwd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.895222 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.895120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.895222 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.895208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.895397 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.895232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.895397 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.895264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.995940 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.995916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.996073 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.995951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfwd\" (UniqueName: \"kubernetes.io/projected/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-kube-api-access-sqfwd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.996073 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.995981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.996073 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.996045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.996251 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.996082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.996251 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.996127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.996473 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.996449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.996593 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.996580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.996817 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.996594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.998396 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.998375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:39.998484 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:39.998464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:40.003051 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:40.003027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfwd\" (UniqueName: \"kubernetes.io/projected/335f0e6d-7d90-4912-8b01-0a3b6c76fe88-kube-api-access-sqfwd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vkxs\" (UID: \"335f0e6d-7d90-4912-8b01-0a3b6c76fe88\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:40.165367 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:40.165307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:40.436266 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:40.436239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"]
Sep 06 05:08:40.437563 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:08:40.437541    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335f0e6d_7d90_4912_8b01_0a3b6c76fe88.slice/crio-5119d0674253d800c56438b6c775112816f28d144fda5bbf7bfd033eeb5269ea WatchSource:0}: Error finding container 5119d0674253d800c56438b6c775112816f28d144fda5bbf7bfd033eeb5269ea: Status 404 returned error can't find the container with id 5119d0674253d800c56438b6c775112816f28d144fda5bbf7bfd033eeb5269ea
Sep 06 05:08:40.802434 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:40.802401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs" event={"ID":"335f0e6d-7d90-4912-8b01-0a3b6c76fe88","Type":"ContainerStarted","Data":"170cd1218c3600c45dc5c8a99324f4ec010528cee712acf3615e7a13aac2bc08"}
Sep 06 05:08:40.802592 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:40.802457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs" event={"ID":"335f0e6d-7d90-4912-8b01-0a3b6c76fe88","Type":"ContainerStarted","Data":"5119d0674253d800c56438b6c775112816f28d144fda5bbf7bfd033eeb5269ea"}
Sep 06 05:08:40.819700 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:40.819650    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs" podStartSLOduration=1.638970784 podStartE2EDuration="1.819635815s" totalImagesPullingTime="180.665031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:08:40.81832807 +0000 UTC m=+796.959888032" watchObservedRunningTime="2026-09-06 05:08:40.819635815 +0000 UTC m=+796.961195757"
Sep 06 05:08:41.039521 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:41.039490    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:08:46.640272 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:46.640244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:08:50.166204 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:50.166175    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:50.166575 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:50.166232    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:50.178534 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:50.178512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:50.866548 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:50.866519    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vkxs"
Sep 06 05:08:54.446911 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:08:54.446875    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:09:17.946801 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:09:17.946756    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:10:10.485449 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.485414    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcd7bc469-r2bf5"]
Sep 06 05:10:10.489027 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.489010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcd7bc469-r2bf5"
Sep 06 05:10:10.492374 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.492349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 06 05:10:10.492561 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.492413    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-phjph\""
Sep 06 05:10:10.494751 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.494726    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcd7bc469-r2bf5"]
Sep 06 05:10:10.537303 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.537273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9195228f-e128-43fa-883d-e1a20a859f34-tls-cert\") pod \"authorino-7bcd7bc469-r2bf5\" (UID: \"9195228f-e128-43fa-883d-e1a20a859f34\") " pod="kuadrant-system/authorino-7bcd7bc469-r2bf5"
Sep 06 05:10:10.537448 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.537387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffxzm\" (UniqueName: \"kubernetes.io/projected/9195228f-e128-43fa-883d-e1a20a859f34-kube-api-access-ffxzm\") pod \"authorino-7bcd7bc469-r2bf5\" (UID: \"9195228f-e128-43fa-883d-e1a20a859f34\") " pod="kuadrant-system/authorino-7bcd7bc469-r2bf5"
Sep 06 05:10:10.638022 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.637982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9195228f-e128-43fa-883d-e1a20a859f34-tls-cert\") pod \"authorino-7bcd7bc469-r2bf5\" (UID: \"9195228f-e128-43fa-883d-e1a20a859f34\") " pod="kuadrant-system/authorino-7bcd7bc469-r2bf5"
Sep 06 05:10:10.638186 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.638042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffxzm\" (UniqueName: \"kubernetes.io/projected/9195228f-e128-43fa-883d-e1a20a859f34-kube-api-access-ffxzm\") pod \"authorino-7bcd7bc469-r2bf5\" (UID: \"9195228f-e128-43fa-883d-e1a20a859f34\") " pod="kuadrant-system/authorino-7bcd7bc469-r2bf5"
Sep 06 05:10:10.640657 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.640637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9195228f-e128-43fa-883d-e1a20a859f34-tls-cert\") pod \"authorino-7bcd7bc469-r2bf5\" (UID: \"9195228f-e128-43fa-883d-e1a20a859f34\") " pod="kuadrant-system/authorino-7bcd7bc469-r2bf5"
Sep 06 05:10:10.645121 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.645101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffxzm\" (UniqueName: \"kubernetes.io/projected/9195228f-e128-43fa-883d-e1a20a859f34-kube-api-access-ffxzm\") pod \"authorino-7bcd7bc469-r2bf5\" (UID: \"9195228f-e128-43fa-883d-e1a20a859f34\") " pod="kuadrant-system/authorino-7bcd7bc469-r2bf5"
Sep 06 05:10:10.819378 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:10.819275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcd7bc469-r2bf5"
Sep 06 05:10:11.090897 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:11.090871    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcd7bc469-r2bf5"]
Sep 06 05:10:11.092664 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:10:11.092635    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9195228f_e128_43fa_883d_e1a20a859f34.slice/crio-0a09290a3cf5c24c5d6a21a56c9983ee30e1a2bcb107eddb41f4dee1f204d3bd WatchSource:0}: Error finding container 0a09290a3cf5c24c5d6a21a56c9983ee30e1a2bcb107eddb41f4dee1f204d3bd: Status 404 returned error can't find the container with id 0a09290a3cf5c24c5d6a21a56c9983ee30e1a2bcb107eddb41f4dee1f204d3bd
Sep 06 05:10:11.281080 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:11.281044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcd7bc469-r2bf5" event={"ID":"9195228f-e128-43fa-883d-e1a20a859f34","Type":"ContainerStarted","Data":"0a09290a3cf5c24c5d6a21a56c9983ee30e1a2bcb107eddb41f4dee1f204d3bd"}
Sep 06 05:10:23.347795 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:23.347754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcd7bc469-r2bf5" event={"ID":"9195228f-e128-43fa-883d-e1a20a859f34","Type":"ContainerStarted","Data":"0b4ae9f2affe77993f06f065528c0c96ea540f7d665ed588a017005d5b7a2d00"}
Sep 06 05:10:23.364323 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:23.364264    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcd7bc469-r2bf5" podStartSLOduration=2.060730667 podStartE2EDuration="13.364247252s" totalImagesPullingTime="11.303516585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:10:23.360843135 +0000 UTC m=+899.502403076" watchObservedRunningTime="2026-09-06 05:10:23.364247252 +0000 UTC m=+899.505807193"
Sep 06 05:10:24.397394 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:24.397363    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:10:24.398222 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:24.398196    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:10:42.461796 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:42.461766    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7561-139988894485680"]
Sep 06 05:10:42.464803 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:42.464788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7561-139988894485680"
Sep 06 05:10:42.467067 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:42.467042    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5l28p\""
Sep 06 05:10:42.468489 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:42.468466    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7561-139988894485680"]
Sep 06 05:10:42.607986 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:42.607960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhvh\" (UniqueName: \"kubernetes.io/projected/512bae4e-fc5e-418a-873a-2bc436cf34e3-kube-api-access-hxhvh\") pod \"test-curl-7561-139988894485680\" (UID: \"512bae4e-fc5e-418a-873a-2bc436cf34e3\") " pod="opendatahub/test-curl-7561-139988894485680"
Sep 06 05:10:42.708399 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:42.708370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhvh\" (UniqueName: \"kubernetes.io/projected/512bae4e-fc5e-418a-873a-2bc436cf34e3-kube-api-access-hxhvh\") pod \"test-curl-7561-139988894485680\" (UID: \"512bae4e-fc5e-418a-873a-2bc436cf34e3\") " pod="opendatahub/test-curl-7561-139988894485680"
Sep 06 05:10:42.718288 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:42.718226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhvh\" (UniqueName: \"kubernetes.io/projected/512bae4e-fc5e-418a-873a-2bc436cf34e3-kube-api-access-hxhvh\") pod \"test-curl-7561-139988894485680\" (UID: \"512bae4e-fc5e-418a-873a-2bc436cf34e3\") " pod="opendatahub/test-curl-7561-139988894485680"
Sep 06 05:10:42.775337 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:42.775308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7561-139988894485680"
Sep 06 05:10:43.044218 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:43.044192    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7561-139988894485680"]
Sep 06 05:10:43.045506 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:10:43.045477    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod512bae4e_fc5e_418a_873a_2bc436cf34e3.slice/crio-a9f58f9829baaea78236dde5409f6afcfafae94324a628ac34a651f01b2a6ded WatchSource:0}: Error finding container a9f58f9829baaea78236dde5409f6afcfafae94324a628ac34a651f01b2a6ded: Status 404 returned error can't find the container with id a9f58f9829baaea78236dde5409f6afcfafae94324a628ac34a651f01b2a6ded
Sep 06 05:10:43.453076 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:43.452997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7561-139988894485680" event={"ID":"512bae4e-fc5e-418a-873a-2bc436cf34e3","Type":"ContainerStarted","Data":"a9f58f9829baaea78236dde5409f6afcfafae94324a628ac34a651f01b2a6ded"}
Sep 06 05:10:44.356614 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:44.356578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:10:45.469103 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:45.469067    2580 generic.go:356] "Generic (PLEG): container finished" podID="512bae4e-fc5e-418a-873a-2bc436cf34e3" containerID="2b42c93eb0e6147252ba0eb03145f1d93b3c5f0b300f608c881ec386dfff63e2" exitCode=0
Sep 06 05:10:45.469501 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:45.469181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7561-139988894485680" event={"ID":"512bae4e-fc5e-418a-873a-2bc436cf34e3","Type":"ContainerDied","Data":"2b42c93eb0e6147252ba0eb03145f1d93b3c5f0b300f608c881ec386dfff63e2"}
Sep 06 05:10:46.603017 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.602992    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7561-139988894485680"
Sep 06 05:10:46.640758 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.640730    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7561-139988894485680_512bae4e-fc5e-418a-873a-2bc436cf34e3/test-curl-7561-139988894485680/0.log"
Sep 06 05:10:46.665186 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.665145    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7561-139988894485680"]
Sep 06 05:10:46.668524 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.668498    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7561-139988894485680"]
Sep 06 05:10:46.748613 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.748534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxhvh\" (UniqueName: \"kubernetes.io/projected/512bae4e-fc5e-418a-873a-2bc436cf34e3-kube-api-access-hxhvh\") pod \"512bae4e-fc5e-418a-873a-2bc436cf34e3\" (UID: \"512bae4e-fc5e-418a-873a-2bc436cf34e3\") "
Sep 06 05:10:46.750895 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.750869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512bae4e-fc5e-418a-873a-2bc436cf34e3-kube-api-access-hxhvh" (OuterVolumeSpecName: "kube-api-access-hxhvh") pod "512bae4e-fc5e-418a-873a-2bc436cf34e3" (UID: "512bae4e-fc5e-418a-873a-2bc436cf34e3"). InnerVolumeSpecName "kube-api-access-hxhvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:10:46.779068 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.779037    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7561-139988894485424"]
Sep 06 05:10:46.779442 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.779427    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512bae4e-fc5e-418a-873a-2bc436cf34e3" containerName="test-curl-7561-139988894485680"
Sep 06 05:10:46.779493 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.779443    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="512bae4e-fc5e-418a-873a-2bc436cf34e3" containerName="test-curl-7561-139988894485680"
Sep 06 05:10:46.779536 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.779528    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="512bae4e-fc5e-418a-873a-2bc436cf34e3" containerName="test-curl-7561-139988894485680"
Sep 06 05:10:46.782652 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.782637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7561-139988894485424"
Sep 06 05:10:46.787448 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.787422    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7561-139988894485424"]
Sep 06 05:10:46.849196 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.849139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k78l2\" (UniqueName: \"kubernetes.io/projected/15249d72-b2f7-46b9-916e-1a32e480c2d2-kube-api-access-k78l2\") pod \"test-curl-7561-139988894485424\" (UID: \"15249d72-b2f7-46b9-916e-1a32e480c2d2\") " pod="opendatahub/test-curl-7561-139988894485424"
Sep 06 05:10:46.849357 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.849300    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxhvh\" (UniqueName: \"kubernetes.io/projected/512bae4e-fc5e-418a-873a-2bc436cf34e3-kube-api-access-hxhvh\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:10:46.949719 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.949677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78l2\" (UniqueName: \"kubernetes.io/projected/15249d72-b2f7-46b9-916e-1a32e480c2d2-kube-api-access-k78l2\") pod \"test-curl-7561-139988894485424\" (UID: \"15249d72-b2f7-46b9-916e-1a32e480c2d2\") " pod="opendatahub/test-curl-7561-139988894485424"
Sep 06 05:10:46.956908 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:46.956888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78l2\" (UniqueName: \"kubernetes.io/projected/15249d72-b2f7-46b9-916e-1a32e480c2d2-kube-api-access-k78l2\") pod \"test-curl-7561-139988894485424\" (UID: \"15249d72-b2f7-46b9-916e-1a32e480c2d2\") " pod="opendatahub/test-curl-7561-139988894485424"
Sep 06 05:10:47.093581 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:47.093495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7561-139988894485424"
Sep 06 05:10:47.362216 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:47.362191    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7561-139988894485424"]
Sep 06 05:10:47.364068 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:10:47.364036    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15249d72_b2f7_46b9_916e_1a32e480c2d2.slice/crio-49fdeb181401b927f3b9503e4196c471cf0415402e03926a7325b23135d405d6 WatchSource:0}: Error finding container 49fdeb181401b927f3b9503e4196c471cf0415402e03926a7325b23135d405d6: Status 404 returned error can't find the container with id 49fdeb181401b927f3b9503e4196c471cf0415402e03926a7325b23135d405d6
Sep 06 05:10:47.480777 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:47.480742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7561-139988894485424" event={"ID":"15249d72-b2f7-46b9-916e-1a32e480c2d2","Type":"ContainerStarted","Data":"49fdeb181401b927f3b9503e4196c471cf0415402e03926a7325b23135d405d6"}
Sep 06 05:10:47.482011 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:47.481994    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9f58f9829baaea78236dde5409f6afcfafae94324a628ac34a651f01b2a6ded"
Sep 06 05:10:47.482150 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:47.482016    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7561-139988894485680"
Sep 06 05:10:47.498261 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:10:47.498229    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7561-139988894485680\" is forbidden: User \"system:node:ip-10-0-128-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-168.ec2.internal' and this object" podUID="512bae4e-fc5e-418a-873a-2bc436cf34e3" pod="opendatahub/test-curl-7561-139988894485680"
Sep 06 05:10:48.348116 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.348081    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="512bae4e-fc5e-418a-873a-2bc436cf34e3" path="/var/lib/kubelet/pods/512bae4e-fc5e-418a-873a-2bc436cf34e3/volumes"
Sep 06 05:10:48.489741 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.489711    2580 generic.go:356] "Generic (PLEG): container finished" podID="15249d72-b2f7-46b9-916e-1a32e480c2d2" containerID="0749413f251d4af08e53e906a460c7823e01dba388bc41a66e9b773cb44a6744" exitCode=0
Sep 06 05:10:48.489887 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.489801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7561-139988894485424" event={"ID":"15249d72-b2f7-46b9-916e-1a32e480c2d2","Type":"ContainerDied","Data":"0749413f251d4af08e53e906a460c7823e01dba388bc41a66e9b773cb44a6744"}
Sep 06 05:10:48.626258 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.626175    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"]
Sep 06 05:10:48.629723 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.629704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.631874 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.631849    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-0cd5b238-kserve-self-signed-certs\""
Sep 06 05:10:48.640101 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.640078    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"]
Sep 06 05:10:48.767130 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.767103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-dshm\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.767308 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.767145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-home\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.767308 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.767186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-tmp-dir\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.767308 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.767203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxjqp\" (UniqueName: \"kubernetes.io/projected/13540bc2-4977-43d2-bab1-ee992450b625-kube-api-access-xxjqp\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.767308 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.767233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-model-cache\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.767308 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.767287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13540bc2-4977-43d2-bab1-ee992450b625-tls-certs\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.868349 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.868313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-dshm\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.868521 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.868359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-home\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.868521 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.868387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-tmp-dir\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.868521 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.868405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxjqp\" (UniqueName: \"kubernetes.io/projected/13540bc2-4977-43d2-bab1-ee992450b625-kube-api-access-xxjqp\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.868521 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.868436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-model-cache\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.868521 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.868457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13540bc2-4977-43d2-bab1-ee992450b625-tls-certs\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.868836 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.868814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-tmp-dir\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.868900 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.868849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-home\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.868900 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.868879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-model-cache\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.870785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.870755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-dshm\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.870910 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.870892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13540bc2-4977-43d2-bab1-ee992450b625-tls-certs\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.875700 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.875680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxjqp\" (UniqueName: \"kubernetes.io/projected/13540bc2-4977-43d2-bab1-ee992450b625-kube-api-access-xxjqp\") pod \"e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") " pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:48.939932 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:48.939856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:49.214127 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.214099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"]
Sep 06 05:10:49.215727 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:10:49.215695    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13540bc2_4977_43d2_bab1_ee992450b625.slice/crio-e40c91193def3da43bcd15530a6e38b853f038b840669f32ca55fba5dc5690d6 WatchSource:0}: Error finding container e40c91193def3da43bcd15530a6e38b853f038b840669f32ca55fba5dc5690d6: Status 404 returned error can't find the container with id e40c91193def3da43bcd15530a6e38b853f038b840669f32ca55fba5dc5690d6
Sep 06 05:10:49.496818 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.496740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd" event={"ID":"13540bc2-4977-43d2-bab1-ee992450b625","Type":"ContainerStarted","Data":"3c56604f4c4862f2ab97d00b61025d37be67d674bac03b9eca4eb1c475a9f1d2"}
Sep 06 05:10:49.496818 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.496781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd" event={"ID":"13540bc2-4977-43d2-bab1-ee992450b625","Type":"ContainerStarted","Data":"e40c91193def3da43bcd15530a6e38b853f038b840669f32ca55fba5dc5690d6"}
Sep 06 05:10:49.512565 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.512514    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd" podStartSLOduration=1.5124935750000001 podStartE2EDuration="1.512493575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:10:49.511927507 +0000 UTC m=+925.653487467" watchObservedRunningTime="2026-09-06 05:10:49.512493575 +0000 UTC m=+925.654053510"
Sep 06 05:10:49.609454 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.609430    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7561-139988894485424"
Sep 06 05:10:49.627310 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.627283    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7561-139988894485424_15249d72-b2f7-46b9-916e-1a32e480c2d2/test-curl-7561-139988894485424/0.log"
Sep 06 05:10:49.649695 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.649664    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7561-139988894485424"]
Sep 06 05:10:49.653005 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.652985    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7561-139988894485424"]
Sep 06 05:10:49.778274 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.778209    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k78l2\" (UniqueName: \"kubernetes.io/projected/15249d72-b2f7-46b9-916e-1a32e480c2d2-kube-api-access-k78l2\") pod \"15249d72-b2f7-46b9-916e-1a32e480c2d2\" (UID: \"15249d72-b2f7-46b9-916e-1a32e480c2d2\") "
Sep 06 05:10:49.780363 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.780343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15249d72-b2f7-46b9-916e-1a32e480c2d2-kube-api-access-k78l2" (OuterVolumeSpecName: "kube-api-access-k78l2") pod "15249d72-b2f7-46b9-916e-1a32e480c2d2" (UID: "15249d72-b2f7-46b9-916e-1a32e480c2d2"). InnerVolumeSpecName "kube-api-access-k78l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:10:49.879048 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:49.879013    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k78l2\" (UniqueName: \"kubernetes.io/projected/15249d72-b2f7-46b9-916e-1a32e480c2d2-kube-api-access-k78l2\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:10:50.348351 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:50.348319    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15249d72-b2f7-46b9-916e-1a32e480c2d2" path="/var/lib/kubelet/pods/15249d72-b2f7-46b9-916e-1a32e480c2d2/volumes"
Sep 06 05:10:50.504837 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:50.504814    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7561-139988894485424"
Sep 06 05:10:50.505308 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:50.504867    2580 scope.go:117] "RemoveContainer" containerID="0749413f251d4af08e53e906a460c7823e01dba388bc41a66e9b773cb44a6744"
Sep 06 05:10:54.461719 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.461686    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"]
Sep 06 05:10:54.462312 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.462285    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15249d72-b2f7-46b9-916e-1a32e480c2d2" containerName="test-curl-7561-139988894485424"
Sep 06 05:10:54.462312 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.462305    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15249d72-b2f7-46b9-916e-1a32e480c2d2" containerName="test-curl-7561-139988894485424"
Sep 06 05:10:54.462434 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.462407    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15249d72-b2f7-46b9-916e-1a32e480c2d2" containerName="test-curl-7561-139988894485424"
Sep 06 05:10:54.466952 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.466929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.469315 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.469295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-9afd7f-serving-cert\""
Sep 06 05:10:54.470594 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.470057    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 05:10:54.470594 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.470125    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 05:10:54.470594 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.470318    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d2kzq\""
Sep 06 05:10:54.470594 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.470067    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 05:10:54.474061 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.474036    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"]
Sep 06 05:10:54.625905 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.625875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-metrics-certs\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.626090 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.625942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-openshift-service-ca\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.626090 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.625963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvwfv\" (UniqueName: \"kubernetes.io/projected/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-kube-api-access-mvwfv\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.626090 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.626009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-maas-api-tls\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.726940 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.726847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-maas-api-tls\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.726940 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.726902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-metrics-certs\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.727210 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.726948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-openshift-service-ca\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.727210 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.726968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwfv\" (UniqueName: \"kubernetes.io/projected/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-kube-api-access-mvwfv\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.727771 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.727737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-openshift-service-ca\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.729641 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.729622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-maas-api-tls\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.729793 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.729771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-metrics-certs\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.734183 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.734140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvwfv\" (UniqueName: \"kubernetes.io/projected/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-kube-api-access-mvwfv\") pod \"maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:54.781811 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:54.781775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:55.054266 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:55.054236    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"]
Sep 06 05:10:55.056777 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:10:55.056748    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6194f7d2_6dbd_48e5_9b73_dc51d1c0a1a3.slice/crio-3e4f697267281c77a6477d8934ad79604e944a54315115b533a5d4813f4314a7 WatchSource:0}: Error finding container 3e4f697267281c77a6477d8934ad79604e944a54315115b533a5d4813f4314a7: Status 404 returned error can't find the container with id 3e4f697267281c77a6477d8934ad79604e944a54315115b533a5d4813f4314a7
Sep 06 05:10:55.538191 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:55.538133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc" event={"ID":"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3","Type":"ContainerStarted","Data":"3e4f697267281c77a6477d8934ad79604e944a54315115b533a5d4813f4314a7"}
Sep 06 05:10:56.551354 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:56.551313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc" event={"ID":"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3","Type":"ContainerStarted","Data":"9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87"}
Sep 06 05:10:56.551828 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:56.551450    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:10:56.567793 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:56.567744    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc" podStartSLOduration=1.284010902 podStartE2EDuration="2.567731753s" totalImagesPullingTime="1.283720851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:10:56.56564527 +0000 UTC m=+932.707205237" watchObservedRunningTime="2026-09-06 05:10:56.567731753 +0000 UTC m=+932.709291694"
Sep 06 05:10:58.940319 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:58.940283    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:58.940705 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:58.940371    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:58.952751 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:58.952729    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:10:59.578429 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:10:59.578403    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:11:02.562690 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:02.562660    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:11:02.949658 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:02.949574    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:11:13.943010 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:13.942969    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"]
Sep 06 05:11:13.952490 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:13.952464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:13.956091 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:13.955602    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9035bc-serving-cert\""
Sep 06 05:11:13.959028 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:13.959001    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"]
Sep 06 05:11:13.989699 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:13.989676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-metrics-certs\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:13.989808 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:13.989742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:13.989808 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:13.989779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wx2f\" (UniqueName: \"kubernetes.io/projected/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-kube-api-access-8wx2f\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:13.989904 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:13.989816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:14.090897 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:14.090871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:14.090897 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:14.090900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wx2f\" (UniqueName: \"kubernetes.io/projected/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-kube-api-access-8wx2f\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:14.091123 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:14.090917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:14.091123 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:14.090997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-metrics-certs\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:14.091123 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:11:14.091017    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-serving-cert: secret "maas-api-e2e-inf-b-9035bc-serving-cert" not found
Sep 06 05:11:14.091123 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:11:14.091083    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-maas-api-tls podName:ab15cb5e-dd12-43eb-8b41-4e310f61a2a4 nodeName:}" failed. No retries permitted until 2026-09-06 05:11:14.591067369 +0000 UTC m=+950.732627289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-maas-api-tls") pod "maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk" (UID: "ab15cb5e-dd12-43eb-8b41-4e310f61a2a4") : secret "maas-api-e2e-inf-b-9035bc-serving-cert" not found
Sep 06 05:11:14.091648 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:14.091626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:14.093541 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:14.093519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-metrics-certs\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:14.099979 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:14.099955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wx2f\" (UniqueName: \"kubernetes.io/projected/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-kube-api-access-8wx2f\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:14.595299 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:14.595268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:14.597715 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:14.597689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:14.875219 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:14.875110    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:15.146808 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:15.146776    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"]
Sep 06 05:11:15.148657 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:11:15.148627    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab15cb5e_dd12_43eb_8b41_4e310f61a2a4.slice/crio-315864e578b3023f4d52401f48179de1ab6dcb299eb4ba7286b156f429e80b5f WatchSource:0}: Error finding container 315864e578b3023f4d52401f48179de1ab6dcb299eb4ba7286b156f429e80b5f: Status 404 returned error can't find the container with id 315864e578b3023f4d52401f48179de1ab6dcb299eb4ba7286b156f429e80b5f
Sep 06 05:11:15.656759 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:15.656725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk" event={"ID":"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4","Type":"ContainerStarted","Data":"33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd"}
Sep 06 05:11:15.656951 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:15.656776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk" event={"ID":"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4","Type":"ContainerStarted","Data":"315864e578b3023f4d52401f48179de1ab6dcb299eb4ba7286b156f429e80b5f"}
Sep 06 05:11:15.656951 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:15.656876    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:15.672854 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:15.672811    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk" podStartSLOduration=2.329397475 podStartE2EDuration="2.67279331s" totalImagesPullingTime="343.395835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:11:15.670854716 +0000 UTC m=+951.812414682" watchObservedRunningTime="2026-09-06 05:11:15.67279331 +0000 UTC m=+951.814353252"
Sep 06 05:11:21.667420 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:21.667383    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:11:25.944792 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:25.944755    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:11:36.556001 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.555965    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"]
Sep 06 05:11:36.561383 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.561362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.563378 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.563354    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-85214949-serving-cert\""
Sep 06 05:11:36.569698 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.569675    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"]
Sep 06 05:11:36.678118 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.678092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-metrics-certs\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.678264 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.678188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-maas-api-tls\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.678264 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.678213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0321e8b-f445-4098-ad04-5e2fbf20a633-openshift-service-ca\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.678264 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.678234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d75vq\" (UniqueName: \"kubernetes.io/projected/e0321e8b-f445-4098-ad04-5e2fbf20a633-kube-api-access-d75vq\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.779653 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.779625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-maas-api-tls\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.779653 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.779652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0321e8b-f445-4098-ad04-5e2fbf20a633-openshift-service-ca\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.779878 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.779674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d75vq\" (UniqueName: \"kubernetes.io/projected/e0321e8b-f445-4098-ad04-5e2fbf20a633-kube-api-access-d75vq\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.779878 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.779814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-metrics-certs\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.780309 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.780289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0321e8b-f445-4098-ad04-5e2fbf20a633-openshift-service-ca\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.782202 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.782181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-metrics-certs\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.782300 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.782257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-maas-api-tls\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.786982 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.786961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d75vq\" (UniqueName: \"kubernetes.io/projected/e0321e8b-f445-4098-ad04-5e2fbf20a633-kube-api-access-d75vq\") pod \"maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:36.874121 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:36.874070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:37.148463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:37.148438    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"]
Sep 06 05:11:37.150406 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:11:37.150380    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0321e8b_f445_4098_ad04_5e2fbf20a633.slice/crio-d8b6444bc9372766955e9ac1beb5b533930f435c71b5c82d628e701d17173ec5 WatchSource:0}: Error finding container d8b6444bc9372766955e9ac1beb5b533930f435c71b5c82d628e701d17173ec5: Status 404 returned error can't find the container with id d8b6444bc9372766955e9ac1beb5b533930f435c71b5c82d628e701d17173ec5
Sep 06 05:11:37.151795 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:37.151780    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 05:11:37.779607 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:37.779522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h" event={"ID":"e0321e8b-f445-4098-ad04-5e2fbf20a633","Type":"ContainerStarted","Data":"c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b"}
Sep 06 05:11:37.779607 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:37.779570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h" event={"ID":"e0321e8b-f445-4098-ad04-5e2fbf20a633","Type":"ContainerStarted","Data":"d8b6444bc9372766955e9ac1beb5b533930f435c71b5c82d628e701d17173ec5"}
Sep 06 05:11:37.779975 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:37.779615    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:37.796036 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:37.795988    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h" podStartSLOduration=1.420007375 podStartE2EDuration="1.795975191s" totalImagesPullingTime="375.967816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:11:37.793419312 +0000 UTC m=+973.934979262" watchObservedRunningTime="2026-09-06 05:11:37.795975191 +0000 UTC m=+973.937535133"
Sep 06 05:11:43.790028 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:43.789996    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:11:48.647780 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:11:48.647744    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:12:07.639567 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.639535    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"]
Sep 06 05:12:07.640088 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.639775    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h" podUID="e0321e8b-f445-4098-ad04-5e2fbf20a633" containerName="maas-api" containerID="cri-o://c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b" gracePeriod=30
Sep 06 05:12:07.892599 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.892531    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:12:07.933475 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.933450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-metrics-certs\") pod \"e0321e8b-f445-4098-ad04-5e2fbf20a633\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") "
Sep 06 05:12:07.933602 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.933479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0321e8b-f445-4098-ad04-5e2fbf20a633-openshift-service-ca\") pod \"e0321e8b-f445-4098-ad04-5e2fbf20a633\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") "
Sep 06 05:12:07.933602 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.933504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-maas-api-tls\") pod \"e0321e8b-f445-4098-ad04-5e2fbf20a633\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") "
Sep 06 05:12:07.933602 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.933563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d75vq\" (UniqueName: \"kubernetes.io/projected/e0321e8b-f445-4098-ad04-5e2fbf20a633-kube-api-access-d75vq\") pod \"e0321e8b-f445-4098-ad04-5e2fbf20a633\" (UID: \"e0321e8b-f445-4098-ad04-5e2fbf20a633\") "
Sep 06 05:12:07.933860 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.933835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0321e8b-f445-4098-ad04-5e2fbf20a633-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0321e8b-f445-4098-ad04-5e2fbf20a633" (UID: "e0321e8b-f445-4098-ad04-5e2fbf20a633"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:12:07.935862 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.935825    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e0321e8b-f445-4098-ad04-5e2fbf20a633" (UID: "e0321e8b-f445-4098-ad04-5e2fbf20a633"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:12:07.935977 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.935869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0321e8b-f445-4098-ad04-5e2fbf20a633" (UID: "e0321e8b-f445-4098-ad04-5e2fbf20a633"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:12:07.935977 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.935883    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0321e8b-f445-4098-ad04-5e2fbf20a633-kube-api-access-d75vq" (OuterVolumeSpecName: "kube-api-access-d75vq") pod "e0321e8b-f445-4098-ad04-5e2fbf20a633" (UID: "e0321e8b-f445-4098-ad04-5e2fbf20a633"). InnerVolumeSpecName "kube-api-access-d75vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:12:07.949865 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.949842    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0321e8b-f445-4098-ad04-5e2fbf20a633" containerID="c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b" exitCode=0
Sep 06 05:12:07.949961 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.949887    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"
Sep 06 05:12:07.949961 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.949927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h" event={"ID":"e0321e8b-f445-4098-ad04-5e2fbf20a633","Type":"ContainerDied","Data":"c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b"}
Sep 06 05:12:07.950074 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.949984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h" event={"ID":"e0321e8b-f445-4098-ad04-5e2fbf20a633","Type":"ContainerDied","Data":"d8b6444bc9372766955e9ac1beb5b533930f435c71b5c82d628e701d17173ec5"}
Sep 06 05:12:07.950074 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.950017    2580 scope.go:117] "RemoveContainer" containerID="c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b"
Sep 06 05:12:07.963812 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.963788    2580 scope.go:117] "RemoveContainer" containerID="c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b"
Sep 06 05:12:07.966210 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:12:07.965634    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b\": container with ID starting with c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b not found: ID does not exist" containerID="c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b"
Sep 06 05:12:07.966210 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.965772    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b"} err="failed to get container status \"c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b\": rpc error: code = NotFound desc = could not find container \"c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b\": container with ID starting with c2a945f7ebd9b70e31b0b08dbbe35f910ec3703f212c883c372170e638965f6b not found: ID does not exist"
Sep 06 05:12:07.977082 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.977062    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"]
Sep 06 05:12:07.980880 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:07.980862    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85214949-5f67c599dd-8sr4h"]
Sep 06 05:12:08.034541 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:08.034524    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:12:08.034622 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:08.034544    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d75vq\" (UniqueName: \"kubernetes.io/projected/e0321e8b-f445-4098-ad04-5e2fbf20a633-kube-api-access-d75vq\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:12:08.034622 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:08.034553    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0321e8b-f445-4098-ad04-5e2fbf20a633-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:12:08.034622 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:08.034562    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0321e8b-f445-4098-ad04-5e2fbf20a633-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:12:08.347627 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:08.347593    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0321e8b-f445-4098-ad04-5e2fbf20a633" path="/var/lib/kubelet/pods/e0321e8b-f445-4098-ad04-5e2fbf20a633/volumes"
Sep 06 05:12:08.541462 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:08.541432    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:12:12.447347 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.447317    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"]
Sep 06 05:12:12.447782 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.447710    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0321e8b-f445-4098-ad04-5e2fbf20a633" containerName="maas-api"
Sep 06 05:12:12.447782 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.447722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0321e8b-f445-4098-ad04-5e2fbf20a633" containerName="maas-api"
Sep 06 05:12:12.447922 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.447791    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0321e8b-f445-4098-ad04-5e2fbf20a633" containerName="maas-api"
Sep 06 05:12:12.452058 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.452043    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.454103 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.454082    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a4f2603a-serving-cert\""
Sep 06 05:12:12.458677 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.458645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"]
Sep 06 05:12:12.577440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.577407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-maas-api-tls\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.577594 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.577447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f75f0ec-0159-44fb-944c-97453633faed-openshift-service-ca\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.577594 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.577524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-metrics-certs\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.577594 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.577578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmzx\" (UniqueName: \"kubernetes.io/projected/8f75f0ec-0159-44fb-944c-97453633faed-kube-api-access-kbmzx\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.678566 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.678533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-metrics-certs\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.678721 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.678582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmzx\" (UniqueName: \"kubernetes.io/projected/8f75f0ec-0159-44fb-944c-97453633faed-kube-api-access-kbmzx\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.678721 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.678628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-maas-api-tls\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.678721 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.678643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f75f0ec-0159-44fb-944c-97453633faed-openshift-service-ca\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.678896 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:12:12.678779    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-serving-cert: secret "maas-api-e2e-ait-a4f2603a-serving-cert" not found
Sep 06 05:12:12.678896 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:12:12.678850    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-maas-api-tls podName:8f75f0ec-0159-44fb-944c-97453633faed nodeName:}" failed. No retries permitted until 2026-09-06 05:12:13.178829462 +0000 UTC m=+1009.320389381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-maas-api-tls") pod "maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc" (UID: "8f75f0ec-0159-44fb-944c-97453633faed") : secret "maas-api-e2e-ait-a4f2603a-serving-cert" not found
Sep 06 05:12:12.679365 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.679347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f75f0ec-0159-44fb-944c-97453633faed-openshift-service-ca\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.681227 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.681204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-metrics-certs\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:12.686672 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:12.686645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmzx\" (UniqueName: \"kubernetes.io/projected/8f75f0ec-0159-44fb-944c-97453633faed-kube-api-access-kbmzx\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:13.184281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:13.184246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-maas-api-tls\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:13.186730 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:13.186700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-maas-api-tls\") pod \"maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:13.363918 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:13.363883    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:13.634004 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:13.633980    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"]
Sep 06 05:12:13.635707 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:12:13.635678    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f75f0ec_0159_44fb_944c_97453633faed.slice/crio-260a830803f9f0e171f1d6a19dbb25588feda1cdb55b06a8317ba17411546ac9 WatchSource:0}: Error finding container 260a830803f9f0e171f1d6a19dbb25588feda1cdb55b06a8317ba17411546ac9: Status 404 returned error can't find the container with id 260a830803f9f0e171f1d6a19dbb25588feda1cdb55b06a8317ba17411546ac9
Sep 06 05:12:13.985592 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:13.985557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc" event={"ID":"8f75f0ec-0159-44fb-944c-97453633faed","Type":"ContainerStarted","Data":"260a830803f9f0e171f1d6a19dbb25588feda1cdb55b06a8317ba17411546ac9"}
Sep 06 05:12:14.993405 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:14.993372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc" event={"ID":"8f75f0ec-0159-44fb-944c-97453633faed","Type":"ContainerStarted","Data":"86c26d8518548254502c1b0485ab696a360c8206de4b2e76117068db4e03af9d"}
Sep 06 05:12:14.993860 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:14.993431    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:15.010446 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:15.010403    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc" podStartSLOduration=2.604359993 podStartE2EDuration="3.010387904s" totalImagesPullingTime="406.027911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:12:15.007557065 +0000 UTC m=+1011.149117006" watchObservedRunningTime="2026-09-06 05:12:15.010387904 +0000 UTC m=+1011.151948238"
Sep 06 05:12:21.004832 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:21.004800    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:27.658449 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:27.658412    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:12:42.977401 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:42.977368    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"]
Sep 06 05:12:42.977936 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:42.977562    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc" podUID="8f75f0ec-0159-44fb-944c-97453633faed" containerName="maas-api" containerID="cri-o://86c26d8518548254502c1b0485ab696a360c8206de4b2e76117068db4e03af9d" gracePeriod=30
Sep 06 05:12:43.173079 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.173039    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f75f0ec-0159-44fb-944c-97453633faed" containerID="86c26d8518548254502c1b0485ab696a360c8206de4b2e76117068db4e03af9d" exitCode=0
Sep 06 05:12:43.173253 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.173083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc" event={"ID":"8f75f0ec-0159-44fb-944c-97453633faed","Type":"ContainerDied","Data":"86c26d8518548254502c1b0485ab696a360c8206de4b2e76117068db4e03af9d"}
Sep 06 05:12:43.218662 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.218637    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:43.331834 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.331767    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-maas-api-tls\") pod \"8f75f0ec-0159-44fb-944c-97453633faed\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") "
Sep 06 05:12:43.331979 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.331833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f75f0ec-0159-44fb-944c-97453633faed-openshift-service-ca\") pod \"8f75f0ec-0159-44fb-944c-97453633faed\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") "
Sep 06 05:12:43.331979 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.331862    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbmzx\" (UniqueName: \"kubernetes.io/projected/8f75f0ec-0159-44fb-944c-97453633faed-kube-api-access-kbmzx\") pod \"8f75f0ec-0159-44fb-944c-97453633faed\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") "
Sep 06 05:12:43.332093 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.331992    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-metrics-certs\") pod \"8f75f0ec-0159-44fb-944c-97453633faed\" (UID: \"8f75f0ec-0159-44fb-944c-97453633faed\") "
Sep 06 05:12:43.332242 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.332210    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f75f0ec-0159-44fb-944c-97453633faed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f75f0ec-0159-44fb-944c-97453633faed" (UID: "8f75f0ec-0159-44fb-944c-97453633faed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:12:43.332365 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.332320    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f75f0ec-0159-44fb-944c-97453633faed-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:12:43.334208 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.334181    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f75f0ec-0159-44fb-944c-97453633faed" (UID: "8f75f0ec-0159-44fb-944c-97453633faed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:12:43.334304 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.334216    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f75f0ec-0159-44fb-944c-97453633faed-kube-api-access-kbmzx" (OuterVolumeSpecName: "kube-api-access-kbmzx") pod "8f75f0ec-0159-44fb-944c-97453633faed" (UID: "8f75f0ec-0159-44fb-944c-97453633faed"). InnerVolumeSpecName "kube-api-access-kbmzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:12:43.334304 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.334243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8f75f0ec-0159-44fb-944c-97453633faed" (UID: "8f75f0ec-0159-44fb-944c-97453633faed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:12:43.433557 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.433534    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:12:43.433557 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.433554    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f75f0ec-0159-44fb-944c-97453633faed-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:12:43.433698 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:43.433563    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmzx\" (UniqueName: \"kubernetes.io/projected/8f75f0ec-0159-44fb-944c-97453633faed-kube-api-access-kbmzx\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:12:44.180061 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:44.180022    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"
Sep 06 05:12:44.180061 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:44.180052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc" event={"ID":"8f75f0ec-0159-44fb-944c-97453633faed","Type":"ContainerDied","Data":"260a830803f9f0e171f1d6a19dbb25588feda1cdb55b06a8317ba17411546ac9"}
Sep 06 05:12:44.180556 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:44.180112    2580 scope.go:117] "RemoveContainer" containerID="86c26d8518548254502c1b0485ab696a360c8206de4b2e76117068db4e03af9d"
Sep 06 05:12:44.206874 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:44.206852    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"]
Sep 06 05:12:44.210671 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:44.210642    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4f2603a-84db4c858d-vlnvc"]
Sep 06 05:12:44.347974 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:44.347942    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f75f0ec-0159-44fb-944c-97453633faed" path="/var/lib/kubelet/pods/8f75f0ec-0159-44fb-944c-97453633faed/volumes"
Sep 06 05:12:47.544883 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:47.544849    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:12:48.151676 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.151641    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"]
Sep 06 05:12:48.152071 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.152056    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f75f0ec-0159-44fb-944c-97453633faed" containerName="maas-api"
Sep 06 05:12:48.152134 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.152073    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f75f0ec-0159-44fb-944c-97453633faed" containerName="maas-api"
Sep 06 05:12:48.152212 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.152141    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f75f0ec-0159-44fb-944c-97453633faed" containerName="maas-api"
Sep 06 05:12:48.155357 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.155340    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.157580 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.157548    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-de5ebc7b-serving-cert\""
Sep 06 05:12:48.165001 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.164980    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"]
Sep 06 05:12:48.172066 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.172043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-maas-api-tls\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.172244 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.172226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-metrics-certs\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.172359 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.172338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12213e6-31d8-401a-b430-20c37608d179-openshift-service-ca\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.172454 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.172440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2pjl\" (UniqueName: \"kubernetes.io/projected/a12213e6-31d8-401a-b430-20c37608d179-kube-api-access-w2pjl\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.273375 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.273348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-maas-api-tls\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.273375 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.273376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-metrics-certs\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.273670 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.273393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12213e6-31d8-401a-b430-20c37608d179-openshift-service-ca\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.273670 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.273455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2pjl\" (UniqueName: \"kubernetes.io/projected/a12213e6-31d8-401a-b430-20c37608d179-kube-api-access-w2pjl\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.274148 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.274125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12213e6-31d8-401a-b430-20c37608d179-openshift-service-ca\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.275915 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.275895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-metrics-certs\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.276016 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.275928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-maas-api-tls\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.281443 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.281423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2pjl\" (UniqueName: \"kubernetes.io/projected/a12213e6-31d8-401a-b430-20c37608d179-kube-api-access-w2pjl\") pod \"maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.468765 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.468746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:48.744338 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:48.744297    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"]
Sep 06 05:12:48.745846 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:12:48.745814    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12213e6_31d8_401a_b430_20c37608d179.slice/crio-ff0aeede20adfa30077e2420b548ec392e74eb5c950f8172267d817447589e7f WatchSource:0}: Error finding container ff0aeede20adfa30077e2420b548ec392e74eb5c950f8172267d817447589e7f: Status 404 returned error can't find the container with id ff0aeede20adfa30077e2420b548ec392e74eb5c950f8172267d817447589e7f
Sep 06 05:12:49.208662 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:49.208632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g" event={"ID":"a12213e6-31d8-401a-b430-20c37608d179","Type":"ContainerStarted","Data":"1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303"}
Sep 06 05:12:49.208777 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:49.208689    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:12:49.208777 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:49.208712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g" event={"ID":"a12213e6-31d8-401a-b430-20c37608d179","Type":"ContainerStarted","Data":"ff0aeede20adfa30077e2420b548ec392e74eb5c950f8172267d817447589e7f"}
Sep 06 05:12:49.226866 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:49.226820    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g" podStartSLOduration=0.877354024 podStartE2EDuration="1.226807191s" totalImagesPullingTime="349.453167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:12:49.224022079 +0000 UTC m=+1045.365582041" watchObservedRunningTime="2026-09-06 05:12:49.226807191 +0000 UTC m=+1045.368367132"
Sep 06 05:12:55.220069 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:12:55.220042    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:13:04.339171 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:04.339118    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:13:15.250374 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:15.250334    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"]
Sep 06 05:13:18.590862 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.590824    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"]
Sep 06 05:13:18.591261 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.591020    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g" podUID="a12213e6-31d8-401a-b430-20c37608d179" containerName="maas-api" containerID="cri-o://1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303" gracePeriod=30
Sep 06 05:13:18.842919 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.842864    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:13:18.924493 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.924467    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-maas-api-tls\") pod \"a12213e6-31d8-401a-b430-20c37608d179\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") "
Sep 06 05:13:18.924670 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.924499    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12213e6-31d8-401a-b430-20c37608d179-openshift-service-ca\") pod \"a12213e6-31d8-401a-b430-20c37608d179\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") "
Sep 06 05:13:18.924670 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.924521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2pjl\" (UniqueName: \"kubernetes.io/projected/a12213e6-31d8-401a-b430-20c37608d179-kube-api-access-w2pjl\") pod \"a12213e6-31d8-401a-b430-20c37608d179\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") "
Sep 06 05:13:18.924670 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.924609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-metrics-certs\") pod \"a12213e6-31d8-401a-b430-20c37608d179\" (UID: \"a12213e6-31d8-401a-b430-20c37608d179\") "
Sep 06 05:13:18.924876 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.924854    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12213e6-31d8-401a-b430-20c37608d179-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a12213e6-31d8-401a-b430-20c37608d179" (UID: "a12213e6-31d8-401a-b430-20c37608d179"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:13:18.924999 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.924981    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12213e6-31d8-401a-b430-20c37608d179-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:18.927043 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.927013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a12213e6-31d8-401a-b430-20c37608d179" (UID: "a12213e6-31d8-401a-b430-20c37608d179"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:13:18.927256 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.927233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a12213e6-31d8-401a-b430-20c37608d179" (UID: "a12213e6-31d8-401a-b430-20c37608d179"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:13:18.927318 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:18.927309    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12213e6-31d8-401a-b430-20c37608d179-kube-api-access-w2pjl" (OuterVolumeSpecName: "kube-api-access-w2pjl") pod "a12213e6-31d8-401a-b430-20c37608d179" (UID: "a12213e6-31d8-401a-b430-20c37608d179"). InnerVolumeSpecName "kube-api-access-w2pjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:13:19.026064 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.026033    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:19.026064 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.026060    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2pjl\" (UniqueName: \"kubernetes.io/projected/a12213e6-31d8-401a-b430-20c37608d179-kube-api-access-w2pjl\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:19.026309 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.026073    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12213e6-31d8-401a-b430-20c37608d179-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:19.383568 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.383541    2580 generic.go:356] "Generic (PLEG): container finished" podID="a12213e6-31d8-401a-b430-20c37608d179" containerID="1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303" exitCode=0
Sep 06 05:13:19.383740 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.383584    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"
Sep 06 05:13:19.383740 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.383621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g" event={"ID":"a12213e6-31d8-401a-b430-20c37608d179","Type":"ContainerDied","Data":"1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303"}
Sep 06 05:13:19.383740 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.383672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g" event={"ID":"a12213e6-31d8-401a-b430-20c37608d179","Type":"ContainerDied","Data":"ff0aeede20adfa30077e2420b548ec392e74eb5c950f8172267d817447589e7f"}
Sep 06 05:13:19.383740 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.383697    2580 scope.go:117] "RemoveContainer" containerID="1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303"
Sep 06 05:13:19.393253 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.393226    2580 scope.go:117] "RemoveContainer" containerID="1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303"
Sep 06 05:13:19.393500 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:13:19.393482    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303\": container with ID starting with 1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303 not found: ID does not exist" containerID="1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303"
Sep 06 05:13:19.393543 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.393508    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303"} err="failed to get container status \"1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303\": rpc error: code = NotFound desc = could not find container \"1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303\": container with ID starting with 1e614b0dd0277f7b8dd992e1aab80f99035254ec7227251306dc2f7d6c01a303 not found: ID does not exist"
Sep 06 05:13:19.404744 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.404719    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"]
Sep 06 05:13:19.406914 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:19.406892    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-de5ebc7b-594f655fd9-tvs7g"]
Sep 06 05:13:20.347315 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:20.347286    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a12213e6-31d8-401a-b430-20c37608d179" path="/var/lib/kubelet/pods/a12213e6-31d8-401a-b430-20c37608d179/volumes"
Sep 06 05:13:20.450116 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:20.450079    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:13:24.247950 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.247912    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"]
Sep 06 05:13:24.248359 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.248343    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12213e6-31d8-401a-b430-20c37608d179" containerName="maas-api"
Sep 06 05:13:24.248359 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.248360    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12213e6-31d8-401a-b430-20c37608d179" containerName="maas-api"
Sep 06 05:13:24.248445 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.248436    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12213e6-31d8-401a-b430-20c37608d179" containerName="maas-api"
Sep 06 05:13:24.251248 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.251232    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.253457 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.253439    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d3a1cb\"/\"kube-root-ca.crt\""
Sep 06 05:13:24.254676 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.254650    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d3a1cb\"/\"openshift-service-ca.crt\""
Sep 06 05:13:24.254843 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.254682    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d3a1cb\"/\"default-dockercfg-rl7tn\""
Sep 06 05:13:24.254843 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.254727    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d3a1cb\"/\"test-model-d3a1cb-kserve-self-signed-certs\""
Sep 06 05:13:24.258625 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.258605    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"]
Sep 06 05:13:24.380236 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.380208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-model-cache\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.380236 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.380239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-dshm\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.380446 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.380258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2nmn\" (UniqueName: \"kubernetes.io/projected/003113de-3c7d-46b9-9d85-59957db225fd-kube-api-access-l2nmn\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.380446 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.380278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-home\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.380446 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.380316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-tmp-dir\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.380446 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.380395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/003113de-3c7d-46b9-9d85-59957db225fd-tls-certs\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.481116 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.481080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-dshm\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.481294 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.481133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2nmn\" (UniqueName: \"kubernetes.io/projected/003113de-3c7d-46b9-9d85-59957db225fd-kube-api-access-l2nmn\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.481294 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.481185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-home\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.481294 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.481212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-tmp-dir\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.481459 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.481293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/003113de-3c7d-46b9-9d85-59957db225fd-tls-certs\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.481459 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.481394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-model-cache\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.481565 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.481501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-home\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.481565 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.481551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-tmp-dir\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.481735 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.481712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-model-cache\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.483404 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.483375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-dshm\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.483789 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.483767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/003113de-3c7d-46b9-9d85-59957db225fd-tls-certs\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.488317 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.488296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2nmn\" (UniqueName: \"kubernetes.io/projected/003113de-3c7d-46b9-9d85-59957db225fd-kube-api-access-l2nmn\") pod \"test-model-d3a1cb-kserve-6bbd7bfdb5-598z5\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") " pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.561380 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.561322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:24.842527 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:24.842501    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"]
Sep 06 05:13:24.844354 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:13:24.844326    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod003113de_3c7d_46b9_9d85_59957db225fd.slice/crio-4e8b7aca37972c5f87951be4d3855b2d8f4a7d0be1af66a326e3e41a42f303c1 WatchSource:0}: Error finding container 4e8b7aca37972c5f87951be4d3855b2d8f4a7d0be1af66a326e3e41a42f303c1: Status 404 returned error can't find the container with id 4e8b7aca37972c5f87951be4d3855b2d8f4a7d0be1af66a326e3e41a42f303c1
Sep 06 05:13:25.420138 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:25.420107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5" event={"ID":"003113de-3c7d-46b9-9d85-59957db225fd","Type":"ContainerStarted","Data":"b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110"}
Sep 06 05:13:25.420541 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:25.420147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5" event={"ID":"003113de-3c7d-46b9-9d85-59957db225fd","Type":"ContainerStarted","Data":"4e8b7aca37972c5f87951be4d3855b2d8f4a7d0be1af66a326e3e41a42f303c1"}
Sep 06 05:13:25.437372 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:25.437325    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5" podStartSLOduration=1.4373085589999999 podStartE2EDuration="1.437308559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:13:25.435284608 +0000 UTC m=+1081.576844551" watchObservedRunningTime="2026-09-06 05:13:25.437308559 +0000 UTC m=+1081.578868500"
Sep 06 05:13:30.272692 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.272632    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd" podUID="13540bc2-4977-43d2-bab1-ee992450b625" containerName="main" containerID="cri-o://3c56604f4c4862f2ab97d00b61025d37be67d674bac03b9eca4eb1c475a9f1d2" gracePeriod=45
Sep 06 05:13:30.449847 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.449814    2580 generic.go:356] "Generic (PLEG): container finished" podID="13540bc2-4977-43d2-bab1-ee992450b625" containerID="3c56604f4c4862f2ab97d00b61025d37be67d674bac03b9eca4eb1c475a9f1d2" exitCode=0
Sep 06 05:13:30.449973 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.449884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd" event={"ID":"13540bc2-4977-43d2-bab1-ee992450b625","Type":"ContainerDied","Data":"3c56604f4c4862f2ab97d00b61025d37be67d674bac03b9eca4eb1c475a9f1d2"}
Sep 06 05:13:30.519337 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.519312    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:13:30.637940 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.637881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxjqp\" (UniqueName: \"kubernetes.io/projected/13540bc2-4977-43d2-bab1-ee992450b625-kube-api-access-xxjqp\") pod \"13540bc2-4977-43d2-bab1-ee992450b625\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") "
Sep 06 05:13:30.637940 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.637935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-model-cache\") pod \"13540bc2-4977-43d2-bab1-ee992450b625\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") "
Sep 06 05:13:30.638143 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.637954    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-dshm\") pod \"13540bc2-4977-43d2-bab1-ee992450b625\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") "
Sep 06 05:13:30.638143 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.637972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-home\") pod \"13540bc2-4977-43d2-bab1-ee992450b625\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") "
Sep 06 05:13:30.638143 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.637986    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-tmp-dir\") pod \"13540bc2-4977-43d2-bab1-ee992450b625\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") "
Sep 06 05:13:30.638143 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.638065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13540bc2-4977-43d2-bab1-ee992450b625-tls-certs\") pod \"13540bc2-4977-43d2-bab1-ee992450b625\" (UID: \"13540bc2-4977-43d2-bab1-ee992450b625\") "
Sep 06 05:13:30.638368 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.638258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-home" (OuterVolumeSpecName: "home") pod "13540bc2-4977-43d2-bab1-ee992450b625" (UID: "13540bc2-4977-43d2-bab1-ee992450b625"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:13:30.638368 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.638280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-model-cache" (OuterVolumeSpecName: "model-cache") pod "13540bc2-4977-43d2-bab1-ee992450b625" (UID: "13540bc2-4977-43d2-bab1-ee992450b625"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:13:30.638368 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.638306    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13540bc2-4977-43d2-bab1-ee992450b625" (UID: "13540bc2-4977-43d2-bab1-ee992450b625"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:13:30.640315 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.640254    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-dshm" (OuterVolumeSpecName: "dshm") pod "13540bc2-4977-43d2-bab1-ee992450b625" (UID: "13540bc2-4977-43d2-bab1-ee992450b625"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:13:30.640419 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.640351    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13540bc2-4977-43d2-bab1-ee992450b625-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13540bc2-4977-43d2-bab1-ee992450b625" (UID: "13540bc2-4977-43d2-bab1-ee992450b625"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:13:30.640419 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.640364    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13540bc2-4977-43d2-bab1-ee992450b625-kube-api-access-xxjqp" (OuterVolumeSpecName: "kube-api-access-xxjqp") pod "13540bc2-4977-43d2-bab1-ee992450b625" (UID: "13540bc2-4977-43d2-bab1-ee992450b625"). InnerVolumeSpecName "kube-api-access-xxjqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:13:30.739005 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.738977    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13540bc2-4977-43d2-bab1-ee992450b625-tls-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:30.739005 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.739003    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxjqp\" (UniqueName: \"kubernetes.io/projected/13540bc2-4977-43d2-bab1-ee992450b625-kube-api-access-xxjqp\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:30.739146 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.739019    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-model-cache\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:30.739146 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.739031    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-dshm\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:30.739146 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.739040    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-home\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:30.739146 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:30.739051    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13540bc2-4977-43d2-bab1-ee992450b625-tmp-dir\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:31.456863 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:31.456835    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"
Sep 06 05:13:31.456863 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:31.456851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd" event={"ID":"13540bc2-4977-43d2-bab1-ee992450b625","Type":"ContainerDied","Data":"e40c91193def3da43bcd15530a6e38b853f038b840669f32ca55fba5dc5690d6"}
Sep 06 05:13:31.457386 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:31.456898    2580 scope.go:117] "RemoveContainer" containerID="3c56604f4c4862f2ab97d00b61025d37be67d674bac03b9eca4eb1c475a9f1d2"
Sep 06 05:13:31.482023 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:31.481994    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"]
Sep 06 05:13:31.487463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:31.487423    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-0cd5b238-kserve-6c47b655f8-txsxd"]
Sep 06 05:13:32.347947 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:32.347918    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13540bc2-4977-43d2-bab1-ee992450b625" path="/var/lib/kubelet/pods/13540bc2-4977-43d2-bab1-ee992450b625/volumes"
Sep 06 05:13:34.562492 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:34.562457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:34.562920 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:34.562543    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:34.575066 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:34.575047    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:35.489555 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:35.489529    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:13:37.542844 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:37.542813    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:13:38.498184 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.495734    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"]
Sep 06 05:13:38.498184 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.496061    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc" podUID="6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3" containerName="maas-api" containerID="cri-o://9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87" gracePeriod=30
Sep 06 05:13:38.747258 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.747236    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:13:38.903994 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.903925    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-maas-api-tls\") pod \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") "
Sep 06 05:13:38.903994 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.903969    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvwfv\" (UniqueName: \"kubernetes.io/projected/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-kube-api-access-mvwfv\") pod \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") "
Sep 06 05:13:38.903994 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.903992    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-openshift-service-ca\") pod \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") "
Sep 06 05:13:38.904256 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.904019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-metrics-certs\") pod \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\" (UID: \"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3\") "
Sep 06 05:13:38.904406 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.904382    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3" (UID: "6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:13:38.906236 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.906212    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-kube-api-access-mvwfv" (OuterVolumeSpecName: "kube-api-access-mvwfv") pod "6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3" (UID: "6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3"). InnerVolumeSpecName "kube-api-access-mvwfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:13:38.906325 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.906243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3" (UID: "6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:13:38.906325 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:38.906284    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3" (UID: "6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:13:39.004580 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.004559    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:39.004580 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.004581    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvwfv\" (UniqueName: \"kubernetes.io/projected/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-kube-api-access-mvwfv\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:39.004751 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.004590    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:39.004751 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.004598    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:13:39.508482 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.508440    2580 generic.go:356] "Generic (PLEG): container finished" podID="6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3" containerID="9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87" exitCode=0
Sep 06 05:13:39.508678 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.508494    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"
Sep 06 05:13:39.508678 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.508637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc" event={"ID":"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3","Type":"ContainerDied","Data":"9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87"}
Sep 06 05:13:39.508787 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.508679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc" event={"ID":"6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3","Type":"ContainerDied","Data":"3e4f697267281c77a6477d8934ad79604e944a54315115b533a5d4813f4314a7"}
Sep 06 05:13:39.508787 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.508707    2580 scope.go:117] "RemoveContainer" containerID="9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87"
Sep 06 05:13:39.521880 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.521863    2580 scope.go:117] "RemoveContainer" containerID="9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87"
Sep 06 05:13:39.522173 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:13:39.522139    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87\": container with ID starting with 9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87 not found: ID does not exist" containerID="9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87"
Sep 06 05:13:39.522236 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.522183    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87"} err="failed to get container status \"9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87\": rpc error: code = NotFound desc = could not find container \"9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87\": container with ID starting with 9b1c26b420bcd441176438d2a35b42fa5898e8c4d753647151ba7259850b7d87 not found: ID does not exist"
Sep 06 05:13:39.531092 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.531071    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"]
Sep 06 05:13:39.534640 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:39.534622    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9afd7f-6d47cd785f-m88gc"]
Sep 06 05:13:40.348125 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:40.348097    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3" path="/var/lib/kubelet/pods/6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3/volumes"
Sep 06 05:13:49.463684 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.463648    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"]
Sep 06 05:13:49.464221 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.464204    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13540bc2-4977-43d2-bab1-ee992450b625" containerName="main"
Sep 06 05:13:49.464282 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.464225    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13540bc2-4977-43d2-bab1-ee992450b625" containerName="main"
Sep 06 05:13:49.464282 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.464254    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3" containerName="maas-api"
Sep 06 05:13:49.464282 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.464265    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3" containerName="maas-api"
Sep 06 05:13:49.464375 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.464369    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="13540bc2-4977-43d2-bab1-ee992450b625" containerName="main"
Sep 06 05:13:49.464413 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.464383    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6194f7d2-6dbd-48e5-9b73-dc51d1c0a1a3" containerName="maas-api"
Sep 06 05:13:49.468883 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.468863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.470965 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.470945    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-17d595-serving-cert\""
Sep 06 05:13:49.476768 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.476747    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"]
Sep 06 05:13:49.481302 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.481277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-maas-api-tls\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.481402 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.481323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-openshift-service-ca\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.481463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.481414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-metrics-certs\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.481463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.481452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fwl\" (UniqueName: \"kubernetes.io/projected/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-kube-api-access-v9fwl\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.582802 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.582770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-metrics-certs\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.582802 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.582804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fwl\" (UniqueName: \"kubernetes.io/projected/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-kube-api-access-v9fwl\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.583068 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.582869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-maas-api-tls\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.583068 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.582888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-openshift-service-ca\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.583068 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:13:49.583001    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-serving-cert: secret "maas-api-e2e-explicit-17d595-serving-cert" not found
Sep 06 05:13:49.583271 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:13:49.583073    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-maas-api-tls podName:89fee8ca-7bcd-4d0f-9538-3b8b31e9923e nodeName:}" failed. No retries permitted until 2026-09-06 05:13:50.083054115 +0000 UTC m=+1106.224614039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-maas-api-tls") pod "maas-api-e2e-explicit-17d595-8546789796-tgpg4" (UID: "89fee8ca-7bcd-4d0f-9538-3b8b31e9923e") : secret "maas-api-e2e-explicit-17d595-serving-cert" not found
Sep 06 05:13:49.583818 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.583706    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-openshift-service-ca\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.585560 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.585543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-metrics-certs\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:49.591692 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:49.591669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fwl\" (UniqueName: \"kubernetes.io/projected/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-kube-api-access-v9fwl\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:50.086769 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:50.086731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-maas-api-tls\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:50.089319 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:50.089289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-maas-api-tls\") pod \"maas-api-e2e-explicit-17d595-8546789796-tgpg4\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:50.381802 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:50.381729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:50.654406 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:50.654374    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"]
Sep 06 05:13:50.655897 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:13:50.655869    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fee8ca_7bcd_4d0f_9538_3b8b31e9923e.slice/crio-0619f446bb1543e7a8dfbff3e3ce56de950bf01b123eb6e652141fb1c9d3a69e WatchSource:0}: Error finding container 0619f446bb1543e7a8dfbff3e3ce56de950bf01b123eb6e652141fb1c9d3a69e: Status 404 returned error can't find the container with id 0619f446bb1543e7a8dfbff3e3ce56de950bf01b123eb6e652141fb1c9d3a69e
Sep 06 05:13:51.573864 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:51.573832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4" event={"ID":"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e","Type":"ContainerStarted","Data":"d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4"}
Sep 06 05:13:51.574027 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:51.573888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4" event={"ID":"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e","Type":"ContainerStarted","Data":"0619f446bb1543e7a8dfbff3e3ce56de950bf01b123eb6e652141fb1c9d3a69e"}
Sep 06 05:13:51.574027 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:51.573913    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:13:51.589757 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:51.589710    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4" podStartSLOduration=2.162158776 podStartE2EDuration="2.589695921s" totalImagesPullingTime="427.537145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:13:51.588688592 +0000 UTC m=+1107.730248533" watchObservedRunningTime="2026-09-06 05:13:51.589695921 +0000 UTC m=+1107.731255916"
Sep 06 05:13:52.456498 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:52.456464    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:13:57.586201 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:13:57.586169    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:14:09.147581 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:09.147545    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:14:24.442297 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:24.442263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:14:40.346791 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:40.346763    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:14:57.844729 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:57.844696    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"]
Sep 06 05:14:57.848337 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:57.848312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:57.851008 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:57.850983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-241a3769-serving-cert\""
Sep 06 05:14:57.857349 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:57.857133    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"]
Sep 06 05:14:57.935303 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:57.935274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-maas-api-tls\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:57.935455 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:57.935308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfv2f\" (UniqueName: \"kubernetes.io/projected/d6904756-84c2-4dba-89d5-ecb5f248752b-kube-api-access-mfv2f\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:57.935455 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:57.935391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-metrics-certs\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:57.935548 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:57.935464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6904756-84c2-4dba-89d5-ecb5f248752b-openshift-service-ca\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:58.036469 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:58.036442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6904756-84c2-4dba-89d5-ecb5f248752b-openshift-service-ca\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:58.036621 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:58.036483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-maas-api-tls\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:58.036621 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:58.036536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfv2f\" (UniqueName: \"kubernetes.io/projected/d6904756-84c2-4dba-89d5-ecb5f248752b-kube-api-access-mfv2f\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:58.036621 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:58.036593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-metrics-certs\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:58.036759 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:14:58.036738    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-serving-cert: secret "maas-api-e2e-mt-241a3769-serving-cert" not found
Sep 06 05:14:58.036828 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:14:58.036817    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-maas-api-tls podName:d6904756-84c2-4dba-89d5-ecb5f248752b nodeName:}" failed. No retries permitted until 2026-09-06 05:14:58.536796658 +0000 UTC m=+1174.678356585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-maas-api-tls") pod "maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p" (UID: "d6904756-84c2-4dba-89d5-ecb5f248752b") : secret "maas-api-e2e-mt-241a3769-serving-cert" not found
Sep 06 05:14:58.037126 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:58.037103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6904756-84c2-4dba-89d5-ecb5f248752b-openshift-service-ca\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:58.039023 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:58.039000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-metrics-certs\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:58.047268 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:58.047246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfv2f\" (UniqueName: \"kubernetes.io/projected/d6904756-84c2-4dba-89d5-ecb5f248752b-kube-api-access-mfv2f\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:58.541312 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:58.541279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-maas-api-tls\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:58.543736 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:58.543707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-maas-api-tls\") pod \"maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:58.760810 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:58.760775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:59.035133 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:59.035108    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"]
Sep 06 05:14:59.036570 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:14:59.036543    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6904756_84c2_4dba_89d5_ecb5f248752b.slice/crio-07a183aae2f3e48dd86608cd9bc4f66e18c28924c696e32321b0bad0d5e3a67f WatchSource:0}: Error finding container 07a183aae2f3e48dd86608cd9bc4f66e18c28924c696e32321b0bad0d5e3a67f: Status 404 returned error can't find the container with id 07a183aae2f3e48dd86608cd9bc4f66e18c28924c696e32321b0bad0d5e3a67f
Sep 06 05:14:59.137267 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:59.137202    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:14:59.954891 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:59.954859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p" event={"ID":"d6904756-84c2-4dba-89d5-ecb5f248752b","Type":"ContainerStarted","Data":"0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031"}
Sep 06 05:14:59.955054 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:59.954911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p" event={"ID":"d6904756-84c2-4dba-89d5-ecb5f248752b","Type":"ContainerStarted","Data":"07a183aae2f3e48dd86608cd9bc4f66e18c28924c696e32321b0bad0d5e3a67f"}
Sep 06 05:14:59.955054 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:59.954935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:14:59.972750 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:14:59.972701    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p" podStartSLOduration=2.558519832 podStartE2EDuration="2.972687736s" totalImagesPullingTime="414.167904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:14:59.969996225 +0000 UTC m=+1176.111556185" watchObservedRunningTime="2026-09-06 05:14:59.972687736 +0000 UTC m=+1176.114247717"
Sep 06 05:15:00.136626 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:00.136600    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2"]
Sep 06 05:15:00.140251 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:00.140224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2"
Sep 06 05:15:00.142408 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:00.142381    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9lqz9\""
Sep 06 05:15:00.148421 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:00.148397    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2"]
Sep 06 05:15:00.154137 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:00.154117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2"
Sep 06 05:15:00.429411 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:00.429385    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2"]
Sep 06 05:15:00.430657 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:15:00.430631    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37bebf4_26e0_4722_a472_7129aa2da95d.slice/crio-ef8fdc5e9f7660c8b4b69f94600fe06cae7cfab51715877213ec61c1698c0bf0 WatchSource:0}: Error finding container ef8fdc5e9f7660c8b4b69f94600fe06cae7cfab51715877213ec61c1698c0bf0: Status 404 returned error can't find the container with id ef8fdc5e9f7660c8b4b69f94600fe06cae7cfab51715877213ec61c1698c0bf0
Sep 06 05:15:00.961388 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:00.961359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2" event={"ID":"b37bebf4-26e0-4722-a472-7129aa2da95d","Type":"ContainerStarted","Data":"ef8fdc5e9f7660c8b4b69f94600fe06cae7cfab51715877213ec61c1698c0bf0"}
Sep 06 05:15:03.982086 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:03.982053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2" event={"ID":"b37bebf4-26e0-4722-a472-7129aa2da95d","Type":"ContainerStarted","Data":"bdbd2f5e61ecc122820f4e9576bbd77147c1983a44870dac3fab8ed07bc84984"}
Sep 06 05:15:03.997815 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:03.997772    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2" podStartSLOduration=1.193397238 podStartE2EDuration="3.997758725s" totalImagesPullingTime="2.804361487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:15:03.995758273 +0000 UTC m=+1180.137318233" watchObservedRunningTime="2026-09-06 05:15:03.997758725 +0000 UTC m=+1180.139318712"
Sep 06 05:15:04.988602 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:04.988565    2580 generic.go:356] "Generic (PLEG): container finished" podID="b37bebf4-26e0-4722-a472-7129aa2da95d" containerID="bdbd2f5e61ecc122820f4e9576bbd77147c1983a44870dac3fab8ed07bc84984" exitCode=0
Sep 06 05:15:04.988984 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:04.988648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2" event={"ID":"b37bebf4-26e0-4722-a472-7129aa2da95d","Type":"ContainerDied","Data":"bdbd2f5e61ecc122820f4e9576bbd77147c1983a44870dac3fab8ed07bc84984"}
Sep 06 05:15:05.966870 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:05.966845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:15:06.128817 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:06.128792    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2"
Sep 06 05:15:07.002143 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:07.002113    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2"
Sep 06 05:15:07.002143 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:07.002124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2" event={"ID":"b37bebf4-26e0-4722-a472-7129aa2da95d","Type":"ContainerDied","Data":"ef8fdc5e9f7660c8b4b69f94600fe06cae7cfab51715877213ec61c1698c0bf0"}
Sep 06 05:15:07.002349 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:07.002186    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef8fdc5e9f7660c8b4b69f94600fe06cae7cfab51715877213ec61c1698c0bf0"
Sep 06 05:15:16.080975 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.080942    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"]
Sep 06 05:15:16.081573 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.081202    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4" podUID="89fee8ca-7bcd-4d0f-9538-3b8b31e9923e" containerName="maas-api" containerID="cri-o://d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4" gracePeriod=30
Sep 06 05:15:16.326355 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.326333    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:15:16.394484 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.394423    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-metrics-certs\") pod \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") "
Sep 06 05:15:16.394484 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.394476    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-openshift-service-ca\") pod \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") "
Sep 06 05:15:16.394648 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.394549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-maas-api-tls\") pod \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") "
Sep 06 05:15:16.394648 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.394611    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9fwl\" (UniqueName: \"kubernetes.io/projected/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-kube-api-access-v9fwl\") pod \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\" (UID: \"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e\") "
Sep 06 05:15:16.394896 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.394861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89fee8ca-7bcd-4d0f-9538-3b8b31e9923e" (UID: "89fee8ca-7bcd-4d0f-9538-3b8b31e9923e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:15:16.396808 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.396786    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89fee8ca-7bcd-4d0f-9538-3b8b31e9923e" (UID: "89fee8ca-7bcd-4d0f-9538-3b8b31e9923e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:15:16.396896 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.396844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-kube-api-access-v9fwl" (OuterVolumeSpecName: "kube-api-access-v9fwl") pod "89fee8ca-7bcd-4d0f-9538-3b8b31e9923e" (UID: "89fee8ca-7bcd-4d0f-9538-3b8b31e9923e"). InnerVolumeSpecName "kube-api-access-v9fwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:15:16.397022 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.397005    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "89fee8ca-7bcd-4d0f-9538-3b8b31e9923e" (UID: "89fee8ca-7bcd-4d0f-9538-3b8b31e9923e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:15:16.496219 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.496194    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:15:16.496355 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.496238    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9fwl\" (UniqueName: \"kubernetes.io/projected/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-kube-api-access-v9fwl\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:15:16.496355 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.496249    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:15:16.496355 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:16.496258    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:15:17.059105 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:17.059071    2580 generic.go:356] "Generic (PLEG): container finished" podID="89fee8ca-7bcd-4d0f-9538-3b8b31e9923e" containerID="d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4" exitCode=0
Sep 06 05:15:17.059289 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:17.059116    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"
Sep 06 05:15:17.059289 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:17.059177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4" event={"ID":"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e","Type":"ContainerDied","Data":"d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4"}
Sep 06 05:15:17.059289 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:17.059235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4" event={"ID":"89fee8ca-7bcd-4d0f-9538-3b8b31e9923e","Type":"ContainerDied","Data":"0619f446bb1543e7a8dfbff3e3ce56de950bf01b123eb6e652141fb1c9d3a69e"}
Sep 06 05:15:17.059289 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:17.059256    2580 scope.go:117] "RemoveContainer" containerID="d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4"
Sep 06 05:15:17.069436 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:17.069419    2580 scope.go:117] "RemoveContainer" containerID="d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4"
Sep 06 05:15:17.069706 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:15:17.069689    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4\": container with ID starting with d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4 not found: ID does not exist" containerID="d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4"
Sep 06 05:15:17.069776 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:17.069715    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4"} err="failed to get container status \"d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4\": rpc error: code = NotFound desc = could not find container \"d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4\": container with ID starting with d20d27672d7b82751e43ab5dc8c00c9d8de62d10bb19ed8d9866544e9b842bd4 not found: ID does not exist"
Sep 06 05:15:17.078894 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:17.078875    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"]
Sep 06 05:15:17.082608 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:17.082587    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-17d595-8546789796-tgpg4"]
Sep 06 05:15:18.347832 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:18.347796    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89fee8ca-7bcd-4d0f-9538-3b8b31e9923e" path="/var/lib/kubelet/pods/89fee8ca-7bcd-4d0f-9538-3b8b31e9923e/volumes"
Sep 06 05:15:21.646439 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:21.646406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:15:24.438766 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:24.438741    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:15:24.440735 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:24.440709    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:15:39.095774 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.095741    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"]
Sep 06 05:15:39.096348 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.095992    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p" podUID="d6904756-84c2-4dba-89d5-ecb5f248752b" containerName="maas-api" containerID="cri-o://0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031" gracePeriod=30
Sep 06 05:15:39.344758 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.344734    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:15:39.499110 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.499081    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfv2f\" (UniqueName: \"kubernetes.io/projected/d6904756-84c2-4dba-89d5-ecb5f248752b-kube-api-access-mfv2f\") pod \"d6904756-84c2-4dba-89d5-ecb5f248752b\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") "
Sep 06 05:15:39.499297 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.499135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-maas-api-tls\") pod \"d6904756-84c2-4dba-89d5-ecb5f248752b\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") "
Sep 06 05:15:39.499297 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.499179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-metrics-certs\") pod \"d6904756-84c2-4dba-89d5-ecb5f248752b\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") "
Sep 06 05:15:39.499394 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.499324    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6904756-84c2-4dba-89d5-ecb5f248752b-openshift-service-ca\") pod \"d6904756-84c2-4dba-89d5-ecb5f248752b\" (UID: \"d6904756-84c2-4dba-89d5-ecb5f248752b\") "
Sep 06 05:15:39.499680 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.499652    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6904756-84c2-4dba-89d5-ecb5f248752b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6904756-84c2-4dba-89d5-ecb5f248752b" (UID: "d6904756-84c2-4dba-89d5-ecb5f248752b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:15:39.501448 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.501427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6904756-84c2-4dba-89d5-ecb5f248752b" (UID: "d6904756-84c2-4dba-89d5-ecb5f248752b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:15:39.501882 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.501855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6904756-84c2-4dba-89d5-ecb5f248752b-kube-api-access-mfv2f" (OuterVolumeSpecName: "kube-api-access-mfv2f") pod "d6904756-84c2-4dba-89d5-ecb5f248752b" (UID: "d6904756-84c2-4dba-89d5-ecb5f248752b"). InnerVolumeSpecName "kube-api-access-mfv2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:15:39.501882 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.501864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6904756-84c2-4dba-89d5-ecb5f248752b" (UID: "d6904756-84c2-4dba-89d5-ecb5f248752b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:15:39.600604 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.600577    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:15:39.600604 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.600599    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6904756-84c2-4dba-89d5-ecb5f248752b-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:15:39.600604 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.600609    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6904756-84c2-4dba-89d5-ecb5f248752b-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:15:39.600782 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:39.600619    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfv2f\" (UniqueName: \"kubernetes.io/projected/d6904756-84c2-4dba-89d5-ecb5f248752b-kube-api-access-mfv2f\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:15:40.196417 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:40.196382    2580 generic.go:356] "Generic (PLEG): container finished" podID="d6904756-84c2-4dba-89d5-ecb5f248752b" containerID="0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031" exitCode=0
Sep 06 05:15:40.196839 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:40.196426    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"
Sep 06 05:15:40.196839 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:40.196457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p" event={"ID":"d6904756-84c2-4dba-89d5-ecb5f248752b","Type":"ContainerDied","Data":"0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031"}
Sep 06 05:15:40.196839 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:40.196527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p" event={"ID":"d6904756-84c2-4dba-89d5-ecb5f248752b","Type":"ContainerDied","Data":"07a183aae2f3e48dd86608cd9bc4f66e18c28924c696e32321b0bad0d5e3a67f"}
Sep 06 05:15:40.196839 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:40.196549    2580 scope.go:117] "RemoveContainer" containerID="0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031"
Sep 06 05:15:40.206706 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:40.206689    2580 scope.go:117] "RemoveContainer" containerID="0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031"
Sep 06 05:15:40.206988 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:15:40.206970    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031\": container with ID starting with 0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031 not found: ID does not exist" containerID="0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031"
Sep 06 05:15:40.207063 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:40.207000    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031"} err="failed to get container status \"0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031\": rpc error: code = NotFound desc = could not find container \"0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031\": container with ID starting with 0719cb3920ca45af244fdfd0450578e3d242e7e0e1349cd4309799f0646db031 not found: ID does not exist"
Sep 06 05:15:40.217408 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:40.217387    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"]
Sep 06 05:15:40.221181 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:40.221145    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-241a3769-8666d4f4d7-pxk4p"]
Sep 06 05:15:40.346921 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:40.346879    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6904756-84c2-4dba-89d5-ecb5f248752b" path="/var/lib/kubelet/pods/d6904756-84c2-4dba-89d5-ecb5f248752b/volumes"
Sep 06 05:15:44.452522 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:44.452489    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:15:50.924910 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:50.924872    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"]
Sep 06 05:15:53.538142 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:15:53.538104    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:16:03.286545 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.286506    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:16:03.355513 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.355478    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"]
Sep 06 05:16:03.355874 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.355860    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6904756-84c2-4dba-89d5-ecb5f248752b" containerName="maas-api"
Sep 06 05:16:03.355928 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.355877    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6904756-84c2-4dba-89d5-ecb5f248752b" containerName="maas-api"
Sep 06 05:16:03.355928 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.355889    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37bebf4-26e0-4722-a472-7129aa2da95d" containerName="cleanup"
Sep 06 05:16:03.355928 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.355897    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37bebf4-26e0-4722-a472-7129aa2da95d" containerName="cleanup"
Sep 06 05:16:03.355928 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.355911    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fee8ca-7bcd-4d0f-9538-3b8b31e9923e" containerName="maas-api"
Sep 06 05:16:03.355928 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.355917    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fee8ca-7bcd-4d0f-9538-3b8b31e9923e" containerName="maas-api"
Sep 06 05:16:03.356090 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.355979    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89fee8ca-7bcd-4d0f-9538-3b8b31e9923e" containerName="maas-api"
Sep 06 05:16:03.356090 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.355988    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37bebf4-26e0-4722-a472-7129aa2da95d" containerName="cleanup"
Sep 06 05:16:03.356090 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.355998    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6904756-84c2-4dba-89d5-ecb5f248752b" containerName="maas-api"
Sep 06 05:16:03.358202 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.358182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.360439 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.360407    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-e24961-serving-cert\""
Sep 06 05:16:03.379036 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.378995    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"]
Sep 06 05:16:03.408021 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.407977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.408260 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.408070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.408260 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.408090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942b1faf-7d5e-4e0c-99a9-d05d493b2635-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.408260 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.408109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns4qd\" (UniqueName: \"kubernetes.io/projected/942b1faf-7d5e-4e0c-99a9-d05d493b2635-kube-api-access-ns4qd\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.509354 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.509311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.509354 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.509361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942b1faf-7d5e-4e0c-99a9-d05d493b2635-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.509633 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.509531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns4qd\" (UniqueName: \"kubernetes.io/projected/942b1faf-7d5e-4e0c-99a9-d05d493b2635-kube-api-access-ns4qd\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.509704 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.509668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.510083 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.510060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942b1faf-7d5e-4e0c-99a9-d05d493b2635-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.512076 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.512054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.512209 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.512191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.516938 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.516916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns4qd\" (UniqueName: \"kubernetes.io/projected/942b1faf-7d5e-4e0c-99a9-d05d493b2635-kube-api-access-ns4qd\") pod \"maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.671745 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.671648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:03.951468 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:03.951441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"]
Sep 06 05:16:03.954219 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:16:03.954190    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942b1faf_7d5e_4e0c_99a9_d05d493b2635.slice/crio-de96bcbd556aa29b7f5c7793bf34d76f67e8e6df98b5c3d8b7a778b309d14a72 WatchSource:0}: Error finding container de96bcbd556aa29b7f5c7793bf34d76f67e8e6df98b5c3d8b7a778b309d14a72: Status 404 returned error can't find the container with id de96bcbd556aa29b7f5c7793bf34d76f67e8e6df98b5c3d8b7a778b309d14a72
Sep 06 05:16:04.102575 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:04.102540    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2"]
Sep 06 05:16:04.105084 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:04.105052    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3a1cb-29811195-5dsz2"]
Sep 06 05:16:04.337019 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:04.336991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp" event={"ID":"942b1faf-7d5e-4e0c-99a9-d05d493b2635","Type":"ContainerStarted","Data":"de96bcbd556aa29b7f5c7793bf34d76f67e8e6df98b5c3d8b7a778b309d14a72"}
Sep 06 05:16:04.347437 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:04.347407    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b37bebf4-26e0-4722-a472-7129aa2da95d" path="/var/lib/kubelet/pods/b37bebf4-26e0-4722-a472-7129aa2da95d/volumes"
Sep 06 05:16:05.345083 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:05.345036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp" event={"ID":"942b1faf-7d5e-4e0c-99a9-d05d493b2635","Type":"ContainerStarted","Data":"c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86"}
Sep 06 05:16:05.345486 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:05.345125    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:05.362421 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:05.362371    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp" podStartSLOduration=2.026553045 podStartE2EDuration="2.362355341s" totalImagesPullingTime="335.802296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:16:05.360472085 +0000 UTC m=+1241.502032045" watchObservedRunningTime="2026-09-06 05:16:05.362355341 +0000 UTC m=+1241.503915323"
Sep 06 05:16:05.947370 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:05.947323    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5" podUID="003113de-3c7d-46b9-9d85-59957db225fd" containerName="main" containerID="cri-o://b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110" gracePeriod=45
Sep 06 05:16:06.198715 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.198648    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:16:06.235364 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.235334    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/003113de-3c7d-46b9-9d85-59957db225fd-tls-certs\") pod \"003113de-3c7d-46b9-9d85-59957db225fd\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") "
Sep 06 05:16:06.235556 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.235372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2nmn\" (UniqueName: \"kubernetes.io/projected/003113de-3c7d-46b9-9d85-59957db225fd-kube-api-access-l2nmn\") pod \"003113de-3c7d-46b9-9d85-59957db225fd\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") "
Sep 06 05:16:06.235556 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.235427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-model-cache\") pod \"003113de-3c7d-46b9-9d85-59957db225fd\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") "
Sep 06 05:16:06.235556 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.235441    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-tmp-dir\") pod \"003113de-3c7d-46b9-9d85-59957db225fd\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") "
Sep 06 05:16:06.235556 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.235478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-dshm\") pod \"003113de-3c7d-46b9-9d85-59957db225fd\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") "
Sep 06 05:16:06.235556 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.235496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-home\") pod \"003113de-3c7d-46b9-9d85-59957db225fd\" (UID: \"003113de-3c7d-46b9-9d85-59957db225fd\") "
Sep 06 05:16:06.235891 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.235780    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "003113de-3c7d-46b9-9d85-59957db225fd" (UID: "003113de-3c7d-46b9-9d85-59957db225fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:16:06.235891 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.235795    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "003113de-3c7d-46b9-9d85-59957db225fd" (UID: "003113de-3c7d-46b9-9d85-59957db225fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:16:06.235891 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.235830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-home" (OuterVolumeSpecName: "home") pod "003113de-3c7d-46b9-9d85-59957db225fd" (UID: "003113de-3c7d-46b9-9d85-59957db225fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:16:06.238075 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.238046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003113de-3c7d-46b9-9d85-59957db225fd-kube-api-access-l2nmn" (OuterVolumeSpecName: "kube-api-access-l2nmn") pod "003113de-3c7d-46b9-9d85-59957db225fd" (UID: "003113de-3c7d-46b9-9d85-59957db225fd"). InnerVolumeSpecName "kube-api-access-l2nmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:16:06.238336 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.238315    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003113de-3c7d-46b9-9d85-59957db225fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "003113de-3c7d-46b9-9d85-59957db225fd" (UID: "003113de-3c7d-46b9-9d85-59957db225fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:16:06.238336 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.238314    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-dshm" (OuterVolumeSpecName: "dshm") pod "003113de-3c7d-46b9-9d85-59957db225fd" (UID: "003113de-3c7d-46b9-9d85-59957db225fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:16:06.337190 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.337136    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/003113de-3c7d-46b9-9d85-59957db225fd-tls-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:16:06.337190 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.337183    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2nmn\" (UniqueName: \"kubernetes.io/projected/003113de-3c7d-46b9-9d85-59957db225fd-kube-api-access-l2nmn\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:16:06.337190 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.337197    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-model-cache\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:16:06.337440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.337209    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-tmp-dir\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:16:06.337440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.337220    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-dshm\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:16:06.337440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.337230    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/003113de-3c7d-46b9-9d85-59957db225fd-home\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:16:06.351723 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.351686    2580 generic.go:356] "Generic (PLEG): container finished" podID="003113de-3c7d-46b9-9d85-59957db225fd" containerID="b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110" exitCode=0
Sep 06 05:16:06.352148 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.351729    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"
Sep 06 05:16:06.352148 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.351768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5" event={"ID":"003113de-3c7d-46b9-9d85-59957db225fd","Type":"ContainerDied","Data":"b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110"}
Sep 06 05:16:06.352148 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.351817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5" event={"ID":"003113de-3c7d-46b9-9d85-59957db225fd","Type":"ContainerDied","Data":"4e8b7aca37972c5f87951be4d3855b2d8f4a7d0be1af66a326e3e41a42f303c1"}
Sep 06 05:16:06.352148 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.351839    2580 scope.go:117] "RemoveContainer" containerID="b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110"
Sep 06 05:16:06.364655 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.364637    2580 scope.go:117] "RemoveContainer" containerID="b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110"
Sep 06 05:16:06.364991 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:16:06.364973    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110\": container with ID starting with b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110 not found: ID does not exist" containerID="b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110"
Sep 06 05:16:06.365031 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.365004    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110"} err="failed to get container status \"b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110\": rpc error: code = NotFound desc = could not find container \"b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110\": container with ID starting with b9ec4c20e670dcbb4e8a63aa83f1b6e50fcfd2e838d3351c71c6a1c22a5ab110 not found: ID does not exist"
Sep 06 05:16:06.373090 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.373056    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"]
Sep 06 05:16:06.375236 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:06.375216    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d3a1cb/test-model-d3a1cb-kserve-6bbd7bfdb5-598z5"]
Sep 06 05:16:08.348209 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:08.348137    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003113de-3c7d-46b9-9d85-59957db225fd" path="/var/lib/kubelet/pods/003113de-3c7d-46b9-9d85-59957db225fd/volumes"
Sep 06 05:16:11.357201 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:11.357142    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:16:14.357240 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:14.357206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:16:21.683851 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:21.683776    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"]
Sep 06 05:16:21.684249 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:21.684086    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk" podUID="ab15cb5e-dd12-43eb-8b41-4e310f61a2a4" containerName="maas-api" containerID="cri-o://33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd" gracePeriod=30
Sep 06 05:16:21.939292 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:21.939235    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:16:22.080638 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.080603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-metrics-certs\") pod \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") "
Sep 06 05:16:22.080832 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.080658    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-openshift-service-ca\") pod \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") "
Sep 06 05:16:22.080832 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.080675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-maas-api-tls\") pod \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") "
Sep 06 05:16:22.080832 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.080700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wx2f\" (UniqueName: \"kubernetes.io/projected/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-kube-api-access-8wx2f\") pod \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\" (UID: \"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4\") "
Sep 06 05:16:22.081037 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.080976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab15cb5e-dd12-43eb-8b41-4e310f61a2a4" (UID: "ab15cb5e-dd12-43eb-8b41-4e310f61a2a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:16:22.083019 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.082986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab15cb5e-dd12-43eb-8b41-4e310f61a2a4" (UID: "ab15cb5e-dd12-43eb-8b41-4e310f61a2a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:16:22.083187 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.083044    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ab15cb5e-dd12-43eb-8b41-4e310f61a2a4" (UID: "ab15cb5e-dd12-43eb-8b41-4e310f61a2a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:16:22.083187 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.083086    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-kube-api-access-8wx2f" (OuterVolumeSpecName: "kube-api-access-8wx2f") pod "ab15cb5e-dd12-43eb-8b41-4e310f61a2a4" (UID: "ab15cb5e-dd12-43eb-8b41-4e310f61a2a4"). InnerVolumeSpecName "kube-api-access-8wx2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:16:22.181646 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.181608    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:16:22.181646 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.181641    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:16:22.181646 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.181652    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:16:22.181646 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.181663    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wx2f\" (UniqueName: \"kubernetes.io/projected/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4-kube-api-access-8wx2f\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:16:22.444201 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.444149    2580 generic.go:356] "Generic (PLEG): container finished" podID="ab15cb5e-dd12-43eb-8b41-4e310f61a2a4" containerID="33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd" exitCode=0
Sep 06 05:16:22.444348 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.444223    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"
Sep 06 05:16:22.444348 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.444242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk" event={"ID":"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4","Type":"ContainerDied","Data":"33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd"}
Sep 06 05:16:22.444348 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.444294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk" event={"ID":"ab15cb5e-dd12-43eb-8b41-4e310f61a2a4","Type":"ContainerDied","Data":"315864e578b3023f4d52401f48179de1ab6dcb299eb4ba7286b156f429e80b5f"}
Sep 06 05:16:22.444348 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.444318    2580 scope.go:117] "RemoveContainer" containerID="33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd"
Sep 06 05:16:22.454085 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.454068    2580 scope.go:117] "RemoveContainer" containerID="33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd"
Sep 06 05:16:22.454378 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:16:22.454359    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd\": container with ID starting with 33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd not found: ID does not exist" containerID="33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd"
Sep 06 05:16:22.454460 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.454386    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd"} err="failed to get container status \"33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd\": rpc error: code = NotFound desc = could not find container \"33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd\": container with ID starting with 33be01655f569b6953e7187be42d2b62f10a6fa59b9e8d3b5cab4f3d7e4ce1fd not found: ID does not exist"
Sep 06 05:16:22.461306 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.461280    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"]
Sep 06 05:16:22.464110 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:22.464087    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9035bc-55cb85465c-cgfhk"]
Sep 06 05:16:24.348411 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:24.348382    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab15cb5e-dd12-43eb-8b41-4e310f61a2a4" path="/var/lib/kubelet/pods/ab15cb5e-dd12-43eb-8b41-4e310f61a2a4/volumes"
Sep 06 05:16:26.745909 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:26.745874    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:16:33.460229 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.460196    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"]
Sep 06 05:16:33.460606 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.460565    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab15cb5e-dd12-43eb-8b41-4e310f61a2a4" containerName="maas-api"
Sep 06 05:16:33.460606 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.460575    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab15cb5e-dd12-43eb-8b41-4e310f61a2a4" containerName="maas-api"
Sep 06 05:16:33.460606 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.460584    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003113de-3c7d-46b9-9d85-59957db225fd" containerName="main"
Sep 06 05:16:33.460606 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.460589    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="003113de-3c7d-46b9-9d85-59957db225fd" containerName="main"
Sep 06 05:16:33.460735 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.460661    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab15cb5e-dd12-43eb-8b41-4e310f61a2a4" containerName="maas-api"
Sep 06 05:16:33.460735 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.460668    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="003113de-3c7d-46b9-9d85-59957db225fd" containerName="main"
Sep 06 05:16:33.464700 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.464680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.467231 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.467208    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-494740-serving-cert\""
Sep 06 05:16:33.474382 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.473908    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"]
Sep 06 05:16:33.586811 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.586781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.586970 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.586825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.586970 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.586934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxlb\" (UniqueName: \"kubernetes.io/projected/5c42e1b0-411e-42f9-99fb-284999098266-kube-api-access-lfxlb\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.587047 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.587035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c42e1b0-411e-42f9-99fb-284999098266-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.688213 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.688181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.688453 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.688231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.688453 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.688282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxlb\" (UniqueName: \"kubernetes.io/projected/5c42e1b0-411e-42f9-99fb-284999098266-kube-api-access-lfxlb\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.688453 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.688335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c42e1b0-411e-42f9-99fb-284999098266-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.688453 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:16:33.688413    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-serving-cert: secret "maas-api-e2e-shared-a-w3-494740-serving-cert" not found
Sep 06 05:16:33.688680 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:16:33.688497    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-maas-api-tls podName:5c42e1b0-411e-42f9-99fb-284999098266 nodeName:}" failed. No retries permitted until 2026-09-06 05:16:34.188477196 +0000 UTC m=+1270.330037116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-maas-api-tls") pod "maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk" (UID: "5c42e1b0-411e-42f9-99fb-284999098266") : secret "maas-api-e2e-shared-a-w3-494740-serving-cert" not found
Sep 06 05:16:33.688969 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.688939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c42e1b0-411e-42f9-99fb-284999098266-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.690771 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.690744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:33.695951 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:33.695929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxlb\" (UniqueName: \"kubernetes.io/projected/5c42e1b0-411e-42f9-99fb-284999098266-kube-api-access-lfxlb\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:34.193661 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:34.193623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:34.196115 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:34.196095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:34.379933 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:34.379896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:34.658712 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:34.658685    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"]
Sep 06 05:16:34.660529 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:16:34.660502    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c42e1b0_411e_42f9_99fb_284999098266.slice/crio-11a1f695ddba4a9c6121700638704da568dee73511b320420783cad4464a16ec WatchSource:0}: Error finding container 11a1f695ddba4a9c6121700638704da568dee73511b320420783cad4464a16ec: Status 404 returned error can't find the container with id 11a1f695ddba4a9c6121700638704da568dee73511b320420783cad4464a16ec
Sep 06 05:16:35.522242 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:35.522202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk" event={"ID":"5c42e1b0-411e-42f9-99fb-284999098266","Type":"ContainerStarted","Data":"43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab"}
Sep 06 05:16:35.522442 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:35.522261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk" event={"ID":"5c42e1b0-411e-42f9-99fb-284999098266","Type":"ContainerStarted","Data":"11a1f695ddba4a9c6121700638704da568dee73511b320420783cad4464a16ec"}
Sep 06 05:16:35.522442 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:35.522327    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:35.541398 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:35.541329    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk" podStartSLOduration=2.204163889 podStartE2EDuration="2.541313836s" totalImagesPullingTime="337.149947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:16:35.540831832 +0000 UTC m=+1271.682391775" watchObservedRunningTime="2026-09-06 05:16:35.541313836 +0000 UTC m=+1271.682873779"
Sep 06 05:16:37.248080 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:37.248046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:16:41.533793 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:41.533766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:16:56.422000 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:56.421964    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7549-140637893817808"]
Sep 06 05:16:56.425585 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:56.425553    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:16:56.427932 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:56.427908    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5l28p\""
Sep 06 05:16:56.431507 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:56.431477    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-140637893817808"]
Sep 06 05:16:56.484178 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:56.484119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v887t\" (UniqueName: \"kubernetes.io/projected/7e7ffb72-f644-4dc6-84e3-6b60df375ff8-kube-api-access-v887t\") pod \"test-curl-7549-140637893817808\" (UID: \"7e7ffb72-f644-4dc6-84e3-6b60df375ff8\") " pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:16:56.584671 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:56.584639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v887t\" (UniqueName: \"kubernetes.io/projected/7e7ffb72-f644-4dc6-84e3-6b60df375ff8-kube-api-access-v887t\") pod \"test-curl-7549-140637893817808\" (UID: \"7e7ffb72-f644-4dc6-84e3-6b60df375ff8\") " pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:16:56.592301 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:56.592274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v887t\" (UniqueName: \"kubernetes.io/projected/7e7ffb72-f644-4dc6-84e3-6b60df375ff8-kube-api-access-v887t\") pod \"test-curl-7549-140637893817808\" (UID: \"7e7ffb72-f644-4dc6-84e3-6b60df375ff8\") " pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:16:56.737802 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:56.737767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:16:57.009485 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:57.009451    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-140637893817808"]
Sep 06 05:16:57.010143 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:16:57.010120    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e7ffb72_f644_4dc6_84e3_6b60df375ff8.slice/crio-55389abc7555143ce91187c3160a83c3d508e22b97c6cb2b53e2be636f29f41b WatchSource:0}: Error finding container 55389abc7555143ce91187c3160a83c3d508e22b97c6cb2b53e2be636f29f41b: Status 404 returned error can't find the container with id 55389abc7555143ce91187c3160a83c3d508e22b97c6cb2b53e2be636f29f41b
Sep 06 05:16:57.011571 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:57.011553    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 05:16:57.644973 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:57.644943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-140637893817808" event={"ID":"7e7ffb72-f644-4dc6-84e3-6b60df375ff8","Type":"ContainerStarted","Data":"55389abc7555143ce91187c3160a83c3d508e22b97c6cb2b53e2be636f29f41b"}
Sep 06 05:16:58.652096 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:58.652058    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e7ffb72-f644-4dc6-84e3-6b60df375ff8" containerID="1ee235b3c472425222446d17ec1684ebbbff24843249c9a15e99f64a337d496e" exitCode=0
Sep 06 05:16:58.652530 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:58.652174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-140637893817808" event={"ID":"7e7ffb72-f644-4dc6-84e3-6b60df375ff8","Type":"ContainerDied","Data":"1ee235b3c472425222446d17ec1684ebbbff24843249c9a15e99f64a337d496e"}
Sep 06 05:16:59.793465 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.793442    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:16:59.825845 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.825819    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7549-140637893817808_7e7ffb72-f644-4dc6-84e3-6b60df375ff8/test-curl-7549-140637893817808/0.log"
Sep 06 05:16:59.848876 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.848828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7549-140637893817808"]
Sep 06 05:16:59.851665 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.851637    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7549-140637893817808"]
Sep 06 05:16:59.916431 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.916351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v887t\" (UniqueName: \"kubernetes.io/projected/7e7ffb72-f644-4dc6-84e3-6b60df375ff8-kube-api-access-v887t\") pod \"7e7ffb72-f644-4dc6-84e3-6b60df375ff8\" (UID: \"7e7ffb72-f644-4dc6-84e3-6b60df375ff8\") "
Sep 06 05:16:59.918561 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.918535    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7ffb72-f644-4dc6-84e3-6b60df375ff8-kube-api-access-v887t" (OuterVolumeSpecName: "kube-api-access-v887t") pod "7e7ffb72-f644-4dc6-84e3-6b60df375ff8" (UID: "7e7ffb72-f644-4dc6-84e3-6b60df375ff8"). InnerVolumeSpecName "kube-api-access-v887t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:16:59.961665 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.961632    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7549-140637893817808"]
Sep 06 05:16:59.962009 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.961996    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e7ffb72-f644-4dc6-84e3-6b60df375ff8" containerName="test-curl-7549-140637893817808"
Sep 06 05:16:59.962058 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.962011    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e7ffb72-f644-4dc6-84e3-6b60df375ff8" containerName="test-curl-7549-140637893817808"
Sep 06 05:16:59.962096 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.962083    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e7ffb72-f644-4dc6-84e3-6b60df375ff8" containerName="test-curl-7549-140637893817808"
Sep 06 05:16:59.964998 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.964983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:16:59.970179 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:16:59.970135    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-140637893817808"]
Sep 06 05:17:00.017015 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:00.016976    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v887t\" (UniqueName: \"kubernetes.io/projected/7e7ffb72-f644-4dc6-84e3-6b60df375ff8-kube-api-access-v887t\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:17:00.117758 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:00.117723    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2rr\" (UniqueName: \"kubernetes.io/projected/f592b07a-e382-4de5-ab70-0884ab9b1837-kube-api-access-9g2rr\") pod \"test-curl-7549-140637893817808\" (UID: \"f592b07a-e382-4de5-ab70-0884ab9b1837\") " pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:17:00.218407 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:00.218377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2rr\" (UniqueName: \"kubernetes.io/projected/f592b07a-e382-4de5-ab70-0884ab9b1837-kube-api-access-9g2rr\") pod \"test-curl-7549-140637893817808\" (UID: \"f592b07a-e382-4de5-ab70-0884ab9b1837\") " pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:17:00.229570 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:00.229538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2rr\" (UniqueName: \"kubernetes.io/projected/f592b07a-e382-4de5-ab70-0884ab9b1837-kube-api-access-9g2rr\") pod \"test-curl-7549-140637893817808\" (UID: \"f592b07a-e382-4de5-ab70-0884ab9b1837\") " pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:17:00.275255 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:00.275228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:17:00.351900 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:00.351865    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e7ffb72-f644-4dc6-84e3-6b60df375ff8" path="/var/lib/kubelet/pods/7e7ffb72-f644-4dc6-84e3-6b60df375ff8/volumes"
Sep 06 05:17:00.545006 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:00.544978    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-140637893817808"]
Sep 06 05:17:00.546811 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:17:00.546784    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf592b07a_e382_4de5_ab70_0884ab9b1837.slice/crio-f01651f7d31f6e00a1d1b8589df95fe8786c7ed38e52d75a53f840fe73f751b1 WatchSource:0}: Error finding container f01651f7d31f6e00a1d1b8589df95fe8786c7ed38e52d75a53f840fe73f751b1: Status 404 returned error can't find the container with id f01651f7d31f6e00a1d1b8589df95fe8786c7ed38e52d75a53f840fe73f751b1
Sep 06 05:17:00.664140 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:00.664104    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:17:00.664334 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:00.664197    2580 scope.go:117] "RemoveContainer" containerID="1ee235b3c472425222446d17ec1684ebbbff24843249c9a15e99f64a337d496e"
Sep 06 05:17:00.665529 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:00.665344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-140637893817808" event={"ID":"f592b07a-e382-4de5-ab70-0884ab9b1837","Type":"ContainerStarted","Data":"f01651f7d31f6e00a1d1b8589df95fe8786c7ed38e52d75a53f840fe73f751b1"}
Sep 06 05:17:01.673043 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:01.672953    2580 generic.go:356] "Generic (PLEG): container finished" podID="f592b07a-e382-4de5-ab70-0884ab9b1837" containerID="d754e1d7630862355225f4f5e6ccda4fe3d24fbf165eca437c3c4521ee679764" exitCode=0
Sep 06 05:17:01.673043 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:01.673005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-140637893817808" event={"ID":"f592b07a-e382-4de5-ab70-0884ab9b1837","Type":"ContainerDied","Data":"d754e1d7630862355225f4f5e6ccda4fe3d24fbf165eca437c3c4521ee679764"}
Sep 06 05:17:02.182489 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.182457    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"]
Sep 06 05:17:02.186786 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.186762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.189011 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.188990    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-167b5b-serving-cert\""
Sep 06 05:17:02.197328 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.196967    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"]
Sep 06 05:17:02.336268 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.336232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4khcr\" (UniqueName: \"kubernetes.io/projected/aae78a95-1862-4947-8a32-6a1ba76ebf98-kube-api-access-4khcr\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.336268 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.336276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.336490 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.336315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.336490 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.336337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae78a95-1862-4947-8a32-6a1ba76ebf98-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.437269 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.437177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4khcr\" (UniqueName: \"kubernetes.io/projected/aae78a95-1862-4947-8a32-6a1ba76ebf98-kube-api-access-4khcr\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.437269 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.437241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.437454 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.437296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.437454 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.437329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae78a95-1862-4947-8a32-6a1ba76ebf98-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.437959 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.437922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae78a95-1862-4947-8a32-6a1ba76ebf98-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.439868 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.439846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.439983 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.439940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.444201 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.444175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4khcr\" (UniqueName: \"kubernetes.io/projected/aae78a95-1862-4947-8a32-6a1ba76ebf98-kube-api-access-4khcr\") pod \"maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.501251 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.501218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:02.778700 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.778670    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"]
Sep 06 05:17:02.780981 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:17:02.780954    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae78a95_1862_4947_8a32_6a1ba76ebf98.slice/crio-b110d5e41e05f9281f2bf002c6b862ba8e44cdc66acbfa2487a59307688e508f WatchSource:0}: Error finding container b110d5e41e05f9281f2bf002c6b862ba8e44cdc66acbfa2487a59307688e508f: Status 404 returned error can't find the container with id b110d5e41e05f9281f2bf002c6b862ba8e44cdc66acbfa2487a59307688e508f
Sep 06 05:17:02.809884 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.809862    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:17:02.832938 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.832918    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7549-140637893817808_f592b07a-e382-4de5-ab70-0884ab9b1837/test-curl-7549-140637893817808/0.log"
Sep 06 05:17:02.855803 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.855777    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7549-140637893817808"]
Sep 06 05:17:02.858532 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.858508    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7549-140637893817808"]
Sep 06 05:17:02.942352 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.942312    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g2rr\" (UniqueName: \"kubernetes.io/projected/f592b07a-e382-4de5-ab70-0884ab9b1837-kube-api-access-9g2rr\") pod \"f592b07a-e382-4de5-ab70-0884ab9b1837\" (UID: \"f592b07a-e382-4de5-ab70-0884ab9b1837\") "
Sep 06 05:17:02.944557 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.944535    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f592b07a-e382-4de5-ab70-0884ab9b1837-kube-api-access-9g2rr" (OuterVolumeSpecName: "kube-api-access-9g2rr") pod "f592b07a-e382-4de5-ab70-0884ab9b1837" (UID: "f592b07a-e382-4de5-ab70-0884ab9b1837"). InnerVolumeSpecName "kube-api-access-9g2rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:17:02.976854 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.976819    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7549-140637893818384"]
Sep 06 05:17:02.977245 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.977230    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f592b07a-e382-4de5-ab70-0884ab9b1837" containerName="test-curl-7549-140637893817808"
Sep 06 05:17:02.977301 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.977246    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f592b07a-e382-4de5-ab70-0884ab9b1837" containerName="test-curl-7549-140637893817808"
Sep 06 05:17:02.977337 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.977332    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f592b07a-e382-4de5-ab70-0884ab9b1837" containerName="test-curl-7549-140637893817808"
Sep 06 05:17:02.980367 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.980352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:02.986082 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:02.986058    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-140637893818384"]
Sep 06 05:17:03.043220 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.043192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hg9v\" (UniqueName: \"kubernetes.io/projected/a1c7d43a-bb99-4c8d-90ae-a552e007e338-kube-api-access-8hg9v\") pod \"test-curl-7549-140637893818384\" (UID: \"a1c7d43a-bb99-4c8d-90ae-a552e007e338\") " pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:03.043366 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.043348    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g2rr\" (UniqueName: \"kubernetes.io/projected/f592b07a-e382-4de5-ab70-0884ab9b1837-kube-api-access-9g2rr\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:17:03.143820 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.143787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hg9v\" (UniqueName: \"kubernetes.io/projected/a1c7d43a-bb99-4c8d-90ae-a552e007e338-kube-api-access-8hg9v\") pod \"test-curl-7549-140637893818384\" (UID: \"a1c7d43a-bb99-4c8d-90ae-a552e007e338\") " pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:03.154232 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.154191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hg9v\" (UniqueName: \"kubernetes.io/projected/a1c7d43a-bb99-4c8d-90ae-a552e007e338-kube-api-access-8hg9v\") pod \"test-curl-7549-140637893818384\" (UID: \"a1c7d43a-bb99-4c8d-90ae-a552e007e338\") " pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:03.291642 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.291559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:03.559590 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.559563    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-140637893818384"]
Sep 06 05:17:03.562278 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:17:03.562250    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1c7d43a_bb99_4c8d_90ae_a552e007e338.slice/crio-10b07e49d96acc13a4e2d1c75760058e9ac8ceb978fcfeaf513e7f364309138a WatchSource:0}: Error finding container 10b07e49d96acc13a4e2d1c75760058e9ac8ceb978fcfeaf513e7f364309138a: Status 404 returned error can't find the container with id 10b07e49d96acc13a4e2d1c75760058e9ac8ceb978fcfeaf513e7f364309138a
Sep 06 05:17:03.690290 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.690257    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:17:03.690290 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.690272    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f01651f7d31f6e00a1d1b8589df95fe8786c7ed38e52d75a53f840fe73f751b1"
Sep 06 05:17:03.692018 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.691991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb" event={"ID":"aae78a95-1862-4947-8a32-6a1ba76ebf98","Type":"ContainerStarted","Data":"e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0"}
Sep 06 05:17:03.692137 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.692048    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:03.692137 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.692067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb" event={"ID":"aae78a95-1862-4947-8a32-6a1ba76ebf98","Type":"ContainerStarted","Data":"b110d5e41e05f9281f2bf002c6b862ba8e44cdc66acbfa2487a59307688e508f"}
Sep 06 05:17:03.693340 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.693319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-140637893818384" event={"ID":"a1c7d43a-bb99-4c8d-90ae-a552e007e338","Type":"ContainerStarted","Data":"10b07e49d96acc13a4e2d1c75760058e9ac8ceb978fcfeaf513e7f364309138a"}
Sep 06 05:17:03.707458 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:17:03.707429    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7549-140637893817808\" is forbidden: User \"system:node:ip-10-0-128-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-168.ec2.internal' and this object" podUID="f592b07a-e382-4de5-ab70-0884ab9b1837" pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:17:03.708264 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:03.708225    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb" podStartSLOduration=1.321921339 podStartE2EDuration="1.708211201s" totalImagesPullingTime="386.289862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:17:03.705755631 +0000 UTC m=+1299.847315573" watchObservedRunningTime="2026-09-06 05:17:03.708211201 +0000 UTC m=+1299.849771143"
Sep 06 05:17:04.346015 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:17:04.345917    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7549-140637893817808\" is forbidden: User \"system:node:ip-10-0-128-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-168.ec2.internal' and this object" podUID="f592b07a-e382-4de5-ab70-0884ab9b1837" pod="opendatahub/test-curl-7549-140637893817808"
Sep 06 05:17:04.347464 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:04.347436    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f592b07a-e382-4de5-ab70-0884ab9b1837" path="/var/lib/kubelet/pods/f592b07a-e382-4de5-ab70-0884ab9b1837/volumes"
Sep 06 05:17:04.700006 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:04.699921    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1c7d43a-bb99-4c8d-90ae-a552e007e338" containerID="77ec118406504e4aa319b1652fa409c129c57c1903bf8d8dc45d7190be37436d" exitCode=0
Sep 06 05:17:04.700184 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:04.700008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-140637893818384" event={"ID":"a1c7d43a-bb99-4c8d-90ae-a552e007e338","Type":"ContainerDied","Data":"77ec118406504e4aa319b1652fa409c129c57c1903bf8d8dc45d7190be37436d"}
Sep 06 05:17:05.856740 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:05.856717    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:05.873790 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:05.873766    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7549-140637893818384_a1c7d43a-bb99-4c8d-90ae-a552e007e338/test-curl-7549-140637893818384/0.log"
Sep 06 05:17:05.896772 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:05.896744    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7549-140637893818384"]
Sep 06 05:17:05.903019 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:05.902995    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7549-140637893818384"]
Sep 06 05:17:05.969566 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:05.969539    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hg9v\" (UniqueName: \"kubernetes.io/projected/a1c7d43a-bb99-4c8d-90ae-a552e007e338-kube-api-access-8hg9v\") pod \"a1c7d43a-bb99-4c8d-90ae-a552e007e338\" (UID: \"a1c7d43a-bb99-4c8d-90ae-a552e007e338\") "
Sep 06 05:17:05.971721 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:05.971697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c7d43a-bb99-4c8d-90ae-a552e007e338-kube-api-access-8hg9v" (OuterVolumeSpecName: "kube-api-access-8hg9v") pod "a1c7d43a-bb99-4c8d-90ae-a552e007e338" (UID: "a1c7d43a-bb99-4c8d-90ae-a552e007e338"). InnerVolumeSpecName "kube-api-access-8hg9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:17:06.015671 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.015644    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7549-140637893818384"]
Sep 06 05:17:06.016060 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.016047    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c7d43a-bb99-4c8d-90ae-a552e007e338" containerName="test-curl-7549-140637893818384"
Sep 06 05:17:06.016104 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.016062    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c7d43a-bb99-4c8d-90ae-a552e007e338" containerName="test-curl-7549-140637893818384"
Sep 06 05:17:06.016137 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.016132    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c7d43a-bb99-4c8d-90ae-a552e007e338" containerName="test-curl-7549-140637893818384"
Sep 06 05:17:06.019080 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.019065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:06.025116 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.025078    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-140637893818384"]
Sep 06 05:17:06.070120 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.070091    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hg9v\" (UniqueName: \"kubernetes.io/projected/a1c7d43a-bb99-4c8d-90ae-a552e007e338-kube-api-access-8hg9v\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:17:06.171144 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.171103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb88p\" (UniqueName: \"kubernetes.io/projected/ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29-kube-api-access-nb88p\") pod \"test-curl-7549-140637893818384\" (UID: \"ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29\") " pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:06.271682 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.271595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb88p\" (UniqueName: \"kubernetes.io/projected/ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29-kube-api-access-nb88p\") pod \"test-curl-7549-140637893818384\" (UID: \"ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29\") " pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:06.278857 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.278832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb88p\" (UniqueName: \"kubernetes.io/projected/ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29-kube-api-access-nb88p\") pod \"test-curl-7549-140637893818384\" (UID: \"ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29\") " pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:06.330892 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.330862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:06.348521 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.348497    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c7d43a-bb99-4c8d-90ae-a552e007e338" path="/var/lib/kubelet/pods/a1c7d43a-bb99-4c8d-90ae-a552e007e338/volumes"
Sep 06 05:17:06.602211 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.602174    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-140637893818384"]
Sep 06 05:17:06.603933 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:17:06.603907    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac5cc1b3_4cb5_4fd8_8b9d_f60dd5e08b29.slice/crio-afea80d016090f9d84b9b1a78fc2b686ce9fbc1f8c0ff5cc47dd9d32e6250e12 WatchSource:0}: Error finding container afea80d016090f9d84b9b1a78fc2b686ce9fbc1f8c0ff5cc47dd9d32e6250e12: Status 404 returned error can't find the container with id afea80d016090f9d84b9b1a78fc2b686ce9fbc1f8c0ff5cc47dd9d32e6250e12
Sep 06 05:17:06.712532 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.712498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-140637893818384" event={"ID":"ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29","Type":"ContainerStarted","Data":"afea80d016090f9d84b9b1a78fc2b686ce9fbc1f8c0ff5cc47dd9d32e6250e12"}
Sep 06 05:17:06.713806 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.713784    2580 scope.go:117] "RemoveContainer" containerID="77ec118406504e4aa319b1652fa409c129c57c1903bf8d8dc45d7190be37436d"
Sep 06 05:17:06.713806 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:06.713800    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:07.720788 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:07.720754    2580 generic.go:356] "Generic (PLEG): container finished" podID="ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29" containerID="cdfb4f23a427e665dab6630b824f7470b763d5a0a6cf8845163a03fc0f6f556e" exitCode=0
Sep 06 05:17:07.721237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:07.720790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-140637893818384" event={"ID":"ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29","Type":"ContainerDied","Data":"cdfb4f23a427e665dab6630b824f7470b763d5a0a6cf8845163a03fc0f6f556e"}
Sep 06 05:17:07.748630 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:07.748597    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:17:08.864580 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:08.864555    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:08.887167 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:08.887129    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7549-140637893818384_ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29/test-curl-7549-140637893818384/0.log"
Sep 06 05:17:08.915458 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:08.915429    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7549-140637893818384"]
Sep 06 05:17:08.918468 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:08.918441    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7549-140637893818384"]
Sep 06 05:17:08.995656 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:08.995570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb88p\" (UniqueName: \"kubernetes.io/projected/ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29-kube-api-access-nb88p\") pod \"ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29\" (UID: \"ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29\") "
Sep 06 05:17:08.997707 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:08.997679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29-kube-api-access-nb88p" (OuterVolumeSpecName: "kube-api-access-nb88p") pod "ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29" (UID: "ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29"). InnerVolumeSpecName "kube-api-access-nb88p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:17:09.097058 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:09.097025    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb88p\" (UniqueName: \"kubernetes.io/projected/ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29-kube-api-access-nb88p\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:17:09.706939 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:09.706907    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:17:09.736245 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:09.736223    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:09.736405 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:09.736255    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afea80d016090f9d84b9b1a78fc2b686ce9fbc1f8c0ff5cc47dd9d32e6250e12"
Sep 06 05:17:09.746434 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:17:09.746404    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7549-140637893818384\" is forbidden: User \"system:node:ip-10-0-128-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-168.ec2.internal' and this object" podUID="ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29" pod="opendatahub/test-curl-7549-140637893818384"
Sep 06 05:17:10.347247 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:10.347215    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29" path="/var/lib/kubelet/pods/ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29/volumes"
Sep 06 05:17:14.366389 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.366356    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"]
Sep 06 05:17:14.366795 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.366755    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29" containerName="test-curl-7549-140637893818384"
Sep 06 05:17:14.366795 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.366768    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29" containerName="test-curl-7549-140637893818384"
Sep 06 05:17:14.366879 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.366842    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5cc1b3-4cb5-4fd8-8b9d-f60dd5e08b29" containerName="test-curl-7549-140637893818384"
Sep 06 05:17:14.370091 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.370075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.372065 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.372036    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1b19002b-serving-cert\""
Sep 06 05:17:14.378902 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.378877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"]
Sep 06 05:17:14.547113 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.547068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.547349 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.547129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb447a8-1870-4566-8dbc-85f1175f2b4d-openshift-service-ca\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.547349 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.547208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-metrics-certs\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.547349 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.547265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hvhh\" (UniqueName: \"kubernetes.io/projected/fbb447a8-1870-4566-8dbc-85f1175f2b4d-kube-api-access-6hvhh\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.648739 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.648637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb447a8-1870-4566-8dbc-85f1175f2b4d-openshift-service-ca\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.648739 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.648690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-metrics-certs\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.648739 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.648708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvhh\" (UniqueName: \"kubernetes.io/projected/fbb447a8-1870-4566-8dbc-85f1175f2b4d-kube-api-access-6hvhh\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.649380 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.648827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.649380 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:17:14.648952    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-serving-cert: secret "maas-api-e2e-mt-1b19002b-serving-cert" not found
Sep 06 05:17:14.649380 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:17:14.649021    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls podName:fbb447a8-1870-4566-8dbc-85f1175f2b4d nodeName:}" failed. No retries permitted until 2026-09-06 05:17:15.149000855 +0000 UTC m=+1311.290560777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls") pod "maas-api-e2e-mt-1b19002b-7474974b77-tr78m" (UID: "fbb447a8-1870-4566-8dbc-85f1175f2b4d") : secret "maas-api-e2e-mt-1b19002b-serving-cert" not found
Sep 06 05:17:14.649562 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.649460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb447a8-1870-4566-8dbc-85f1175f2b4d-openshift-service-ca\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.651560 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.651540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-metrics-certs\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:14.659535 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:14.659509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hvhh\" (UniqueName: \"kubernetes.io/projected/fbb447a8-1870-4566-8dbc-85f1175f2b4d-kube-api-access-6hvhh\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:15.156336 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:15.156286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:15.156507 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:17:15.156401    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-serving-cert: secret "maas-api-e2e-mt-1b19002b-serving-cert" not found
Sep 06 05:17:15.156507 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:17:15.156479    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls podName:fbb447a8-1870-4566-8dbc-85f1175f2b4d nodeName:}" failed. No retries permitted until 2026-09-06 05:17:16.1564639 +0000 UTC m=+1312.298023820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls") pod "maas-api-e2e-mt-1b19002b-7474974b77-tr78m" (UID: "fbb447a8-1870-4566-8dbc-85f1175f2b4d") : secret "maas-api-e2e-mt-1b19002b-serving-cert" not found
Sep 06 05:17:16.166458 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:16.166409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:16.169137 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:16.169109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls\") pod \"maas-api-e2e-mt-1b19002b-7474974b77-tr78m\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:16.183122 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:16.183094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:16.457672 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:16.457646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"]
Sep 06 05:17:16.459398 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:17:16.459359    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb447a8_1870_4566_8dbc_85f1175f2b4d.slice/crio-e77be5702cfc8304edebe9b556453f3b544140ac4097c54614b15dc5007c60a5 WatchSource:0}: Error finding container e77be5702cfc8304edebe9b556453f3b544140ac4097c54614b15dc5007c60a5: Status 404 returned error can't find the container with id e77be5702cfc8304edebe9b556453f3b544140ac4097c54614b15dc5007c60a5
Sep 06 05:17:16.778111 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:16.778069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m" event={"ID":"fbb447a8-1870-4566-8dbc-85f1175f2b4d","Type":"ContainerStarted","Data":"e77be5702cfc8304edebe9b556453f3b544140ac4097c54614b15dc5007c60a5"}
Sep 06 05:17:17.784825 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:17.784790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m" event={"ID":"fbb447a8-1870-4566-8dbc-85f1175f2b4d","Type":"ContainerStarted","Data":"1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1"}
Sep 06 05:17:17.785237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:17.784908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:17.801782 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:17.801730    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m" podStartSLOduration=3.444579658 podStartE2EDuration="3.801714211s" totalImagesPullingTime="357.134553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:17:17.799798167 +0000 UTC m=+1313.941358130" watchObservedRunningTime="2026-09-06 05:17:17.801714211 +0000 UTC m=+1313.943274198"
Sep 06 05:17:21.941213 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:21.941170    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:17:23.796210 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:23.796180    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:17:24.468955 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:24.468921    2580 scope.go:117] "RemoveContainer" containerID="2b42c93eb0e6147252ba0eb03145f1d93b3c5f0b300f608c881ec386dfff63e2"
Sep 06 05:17:31.047220 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.047182    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"]
Sep 06 05:17:31.052685 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.052659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.056191 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.056096    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-494740\"/\"kube-root-ca.crt\""
Sep 06 05:17:31.056191 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.056134    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-494740\"/\"default-dockercfg-7h4kg\""
Sep 06 05:17:31.056424 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.056396    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-494740\"/\"openshift-service-ca.crt\""
Sep 06 05:17:31.056642 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.056624    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-494740\"/\"auth-test-model-494740-kserve-self-signed-certs\""
Sep 06 05:17:31.057595 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.057566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"]
Sep 06 05:17:31.094141 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.094102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-dshm\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.094332 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.094145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-home\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.094332 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.094225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tls-certs\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.094332 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.094246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-model-cache\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.094332 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.094269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tmp-dir\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.094332 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.094308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfx5x\" (UniqueName: \"kubernetes.io/projected/6d27d33b-eee8-4869-98d1-67e4e136ebf3-kube-api-access-zfx5x\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.195664 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.195625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-dshm\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.195664 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.195668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-home\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.195892 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.195707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tls-certs\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.195892 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.195728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-model-cache\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.195892 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.195766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tmp-dir\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.195892 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.195813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfx5x\" (UniqueName: \"kubernetes.io/projected/6d27d33b-eee8-4869-98d1-67e4e136ebf3-kube-api-access-zfx5x\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.196172 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.196124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-model-cache\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.196293 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.196206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tmp-dir\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.196381 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.196364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-home\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.198308 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.198286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-dshm\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.198599 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.198577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tls-certs\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.203117 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.203093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfx5x\" (UniqueName: \"kubernetes.io/projected/6d27d33b-eee8-4869-98d1-67e4e136ebf3-kube-api-access-zfx5x\") pod \"auth-test-model-494740-kserve-79dd7db79f-27zcn\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") " pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.364616 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.364522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:31.646001 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.645973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"]
Sep 06 05:17:31.647948 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:17:31.647914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d27d33b_eee8_4869_98d1_67e4e136ebf3.slice/crio-e20c18ca662656f8ea86ca7052cf0760b6025fa5ea1c1b7e377b659aebe1c6a5 WatchSource:0}: Error finding container e20c18ca662656f8ea86ca7052cf0760b6025fa5ea1c1b7e377b659aebe1c6a5: Status 404 returned error can't find the container with id e20c18ca662656f8ea86ca7052cf0760b6025fa5ea1c1b7e377b659aebe1c6a5
Sep 06 05:17:31.869060 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.869020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn" event={"ID":"6d27d33b-eee8-4869-98d1-67e4e136ebf3","Type":"ContainerStarted","Data":"f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282"}
Sep 06 05:17:31.869281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.869075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn" event={"ID":"6d27d33b-eee8-4869-98d1-67e4e136ebf3","Type":"ContainerStarted","Data":"e20c18ca662656f8ea86ca7052cf0760b6025fa5ea1c1b7e377b659aebe1c6a5"}
Sep 06 05:17:31.885769 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:31.885711    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn" podStartSLOduration=0.885693331 podStartE2EDuration="885.693331ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:17:31.882956692 +0000 UTC m=+1328.024516633" watchObservedRunningTime="2026-09-06 05:17:31.885693331 +0000 UTC m=+1328.027253322"
Sep 06 05:17:34.244486 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:34.244441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:17:41.365218 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:41.365182    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:41.365612 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:41.365231    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:41.377695 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:41.377666    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:41.937080 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:41.937053    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:17:48.848714 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.848679    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"]
Sep 06 05:17:48.852983 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.852961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:48.855940 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.855914    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1b19002b\"/\"default-dockercfg-bs27q\""
Sep 06 05:17:48.856096 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.855986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1b19002b\"/\"openshift-service-ca.crt\""
Sep 06 05:17:48.856213 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.856090    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1b19002b\"/\"e2e-lifecycle-model-1b19002b-kserve-self-signed-certs\""
Sep 06 05:17:48.856213 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.856208    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1b19002b\"/\"kube-root-ca.crt\""
Sep 06 05:17:48.860498 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.860476    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"]
Sep 06 05:17:48.962899 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.962872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-model-cache\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:48.963054 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.962912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lf5\" (UniqueName: \"kubernetes.io/projected/f3c9add5-e317-4e49-8702-b86e98da66c7-kube-api-access-n5lf5\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:48.963054 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.962943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-home\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:48.963054 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.963004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-tmp-dir\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:48.963054 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.963036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-dshm\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:48.963259 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:48.963118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c9add5-e317-4e49-8702-b86e98da66c7-tls-certs\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.063614 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.063584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-dshm\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.063779 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.063645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c9add5-e317-4e49-8702-b86e98da66c7-tls-certs\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.063779 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.063693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-model-cache\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.063779 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.063712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lf5\" (UniqueName: \"kubernetes.io/projected/f3c9add5-e317-4e49-8702-b86e98da66c7-kube-api-access-n5lf5\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.063779 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.063730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-home\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.063779 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.063754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-tmp-dir\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.064079 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.064056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-model-cache\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.064148 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.064120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-tmp-dir\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.064222 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.064126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-home\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.066115 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.066091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-dshm\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.066234 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.066218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c9add5-e317-4e49-8702-b86e98da66c7-tls-certs\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.070925 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.070903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lf5\" (UniqueName: \"kubernetes.io/projected/f3c9add5-e317-4e49-8702-b86e98da66c7-kube-api-access-n5lf5\") pod \"e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") " pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.165026 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.164966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:49.439354 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.439330    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"]
Sep 06 05:17:49.440964 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:17:49.440933    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3c9add5_e317_4e49_8702_b86e98da66c7.slice/crio-707c321531cd4be6a92de7faa0a7db79006a73184280a54da0dc367d30dd7b4b WatchSource:0}: Error finding container 707c321531cd4be6a92de7faa0a7db79006a73184280a54da0dc367d30dd7b4b: Status 404 returned error can't find the container with id 707c321531cd4be6a92de7faa0a7db79006a73184280a54da0dc367d30dd7b4b
Sep 06 05:17:49.971562 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.971520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq" event={"ID":"f3c9add5-e317-4e49-8702-b86e98da66c7","Type":"ContainerStarted","Data":"3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384"}
Sep 06 05:17:49.971562 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.971563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq" event={"ID":"f3c9add5-e317-4e49-8702-b86e98da66c7","Type":"ContainerStarted","Data":"707c321531cd4be6a92de7faa0a7db79006a73184280a54da0dc367d30dd7b4b"}
Sep 06 05:17:49.987834 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:49.987785    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq" podStartSLOduration=1.987771189 podStartE2EDuration="1.987771189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:17:49.985294508 +0000 UTC m=+1346.126854471" watchObservedRunningTime="2026-09-06 05:17:49.987771189 +0000 UTC m=+1346.129331130"
Sep 06 05:17:55.645212 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:55.645172    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:17:59.165772 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:59.165731    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:59.166269 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:59.165794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:17:59.178392 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:17:59.178366    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:18:00.048854 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:00.048827    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:18:14.845104 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:14.842565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:18:29.751344 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:29.751306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:18:45.343573 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:45.343542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:18:48.633529 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:48.633498    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"]
Sep 06 05:18:48.634108 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:48.633714    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m" podUID="fbb447a8-1870-4566-8dbc-85f1175f2b4d" containerName="maas-api" containerID="cri-o://1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1" gracePeriod=30
Sep 06 05:18:48.791802 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:48.791763    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m" podUID="fbb447a8-1870-4566-8dbc-85f1175f2b4d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.64:8443/health\": dial tcp 10.132.0.64:8443: connect: connection refused"
Sep 06 05:18:48.887517 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:48.887457    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:18:49.020882 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.020778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls\") pod \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") "
Sep 06 05:18:49.021103 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.020948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hvhh\" (UniqueName: \"kubernetes.io/projected/fbb447a8-1870-4566-8dbc-85f1175f2b4d-kube-api-access-6hvhh\") pod \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") "
Sep 06 05:18:49.021103 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.020988    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb447a8-1870-4566-8dbc-85f1175f2b4d-openshift-service-ca\") pod \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") "
Sep 06 05:18:49.021103 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.021066    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-metrics-certs\") pod \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\" (UID: \"fbb447a8-1870-4566-8dbc-85f1175f2b4d\") "
Sep 06 05:18:49.021545 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.021510    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb447a8-1870-4566-8dbc-85f1175f2b4d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fbb447a8-1870-4566-8dbc-85f1175f2b4d" (UID: "fbb447a8-1870-4566-8dbc-85f1175f2b4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:18:49.023644 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.023617    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fbb447a8-1870-4566-8dbc-85f1175f2b4d" (UID: "fbb447a8-1870-4566-8dbc-85f1175f2b4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:18:49.023832 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.023795    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fbb447a8-1870-4566-8dbc-85f1175f2b4d" (UID: "fbb447a8-1870-4566-8dbc-85f1175f2b4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:18:49.023939 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.023856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb447a8-1870-4566-8dbc-85f1175f2b4d-kube-api-access-6hvhh" (OuterVolumeSpecName: "kube-api-access-6hvhh") pod "fbb447a8-1870-4566-8dbc-85f1175f2b4d" (UID: "fbb447a8-1870-4566-8dbc-85f1175f2b4d"). InnerVolumeSpecName "kube-api-access-6hvhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:18:49.122326 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.122293    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:18:49.122326 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.122320    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hvhh\" (UniqueName: \"kubernetes.io/projected/fbb447a8-1870-4566-8dbc-85f1175f2b4d-kube-api-access-6hvhh\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:18:49.122326 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.122329    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb447a8-1870-4566-8dbc-85f1175f2b4d-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:18:49.122562 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.122338    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb447a8-1870-4566-8dbc-85f1175f2b4d-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:18:49.326184 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.326119    2580 generic.go:356] "Generic (PLEG): container finished" podID="fbb447a8-1870-4566-8dbc-85f1175f2b4d" containerID="1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1" exitCode=0
Sep 06 05:18:49.326388 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.326189    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"
Sep 06 05:18:49.326388 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.326197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m" event={"ID":"fbb447a8-1870-4566-8dbc-85f1175f2b4d","Type":"ContainerDied","Data":"1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1"}
Sep 06 05:18:49.326388 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.326252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m" event={"ID":"fbb447a8-1870-4566-8dbc-85f1175f2b4d","Type":"ContainerDied","Data":"e77be5702cfc8304edebe9b556453f3b544140ac4097c54614b15dc5007c60a5"}
Sep 06 05:18:49.326388 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.326280    2580 scope.go:117] "RemoveContainer" containerID="1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1"
Sep 06 05:18:49.336935 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.336918    2580 scope.go:117] "RemoveContainer" containerID="1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1"
Sep 06 05:18:49.337223 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:18:49.337205    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1\": container with ID starting with 1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1 not found: ID does not exist" containerID="1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1"
Sep 06 05:18:49.337294 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.337230    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1"} err="failed to get container status \"1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1\": rpc error: code = NotFound desc = could not find container \"1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1\": container with ID starting with 1edda1c81e700d3bd373c9c0a321a2176fb9a85d224b8ec3acd0382a5ae2c4c1 not found: ID does not exist"
Sep 06 05:18:49.346889 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.346868    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"]
Sep 06 05:18:49.350433 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:49.350416    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b19002b-7474974b77-tr78m"]
Sep 06 05:18:50.347421 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:50.347379    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbb447a8-1870-4566-8dbc-85f1175f2b4d" path="/var/lib/kubelet/pods/fbb447a8-1870-4566-8dbc-85f1175f2b4d/volumes"
Sep 06 05:18:56.952141 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:18:56.952111    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"]
Sep 06 05:19:03.740208 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:03.740173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:19:11.972543 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:11.972495    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq" podUID="f3c9add5-e317-4e49-8702-b86e98da66c7" containerName="main" containerID="cri-o://3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384" gracePeriod=45
Sep 06 05:19:12.226387 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.226328    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:19:12.314871 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.314843    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-dshm\") pod \"f3c9add5-e317-4e49-8702-b86e98da66c7\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") "
Sep 06 05:19:12.315038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.314895    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-home\") pod \"f3c9add5-e317-4e49-8702-b86e98da66c7\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") "
Sep 06 05:19:12.315038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.314943    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-model-cache\") pod \"f3c9add5-e317-4e49-8702-b86e98da66c7\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") "
Sep 06 05:19:12.315038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.314960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c9add5-e317-4e49-8702-b86e98da66c7-tls-certs\") pod \"f3c9add5-e317-4e49-8702-b86e98da66c7\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") "
Sep 06 05:19:12.315038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.314983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-tmp-dir\") pod \"f3c9add5-e317-4e49-8702-b86e98da66c7\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") "
Sep 06 05:19:12.315038 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.315012    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5lf5\" (UniqueName: \"kubernetes.io/projected/f3c9add5-e317-4e49-8702-b86e98da66c7-kube-api-access-n5lf5\") pod \"f3c9add5-e317-4e49-8702-b86e98da66c7\" (UID: \"f3c9add5-e317-4e49-8702-b86e98da66c7\") "
Sep 06 05:19:12.315388 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.315245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3c9add5-e317-4e49-8702-b86e98da66c7" (UID: "f3c9add5-e317-4e49-8702-b86e98da66c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:19:12.315388 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.315260    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-home" (OuterVolumeSpecName: "home") pod "f3c9add5-e317-4e49-8702-b86e98da66c7" (UID: "f3c9add5-e317-4e49-8702-b86e98da66c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:19:12.315388 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.315328    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3c9add5-e317-4e49-8702-b86e98da66c7" (UID: "f3c9add5-e317-4e49-8702-b86e98da66c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:19:12.315542 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.315447    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-home\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:19:12.315542 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.315465    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-model-cache\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:19:12.315542 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.315480    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-tmp-dir\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:19:12.317240 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.317205    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-dshm" (OuterVolumeSpecName: "dshm") pod "f3c9add5-e317-4e49-8702-b86e98da66c7" (UID: "f3c9add5-e317-4e49-8702-b86e98da66c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:19:12.317345 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.317296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c9add5-e317-4e49-8702-b86e98da66c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3c9add5-e317-4e49-8702-b86e98da66c7" (UID: "f3c9add5-e317-4e49-8702-b86e98da66c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:19:12.317345 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.317305    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c9add5-e317-4e49-8702-b86e98da66c7-kube-api-access-n5lf5" (OuterVolumeSpecName: "kube-api-access-n5lf5") pod "f3c9add5-e317-4e49-8702-b86e98da66c7" (UID: "f3c9add5-e317-4e49-8702-b86e98da66c7"). InnerVolumeSpecName "kube-api-access-n5lf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:19:12.416580 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.416546    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c9add5-e317-4e49-8702-b86e98da66c7-dshm\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:19:12.416580 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.416577    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c9add5-e317-4e49-8702-b86e98da66c7-tls-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:19:12.416825 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.416593    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lf5\" (UniqueName: \"kubernetes.io/projected/f3c9add5-e317-4e49-8702-b86e98da66c7-kube-api-access-n5lf5\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:19:12.463749 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.463717    2580 generic.go:356] "Generic (PLEG): container finished" podID="f3c9add5-e317-4e49-8702-b86e98da66c7" containerID="3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384" exitCode=0
Sep 06 05:19:12.463902 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.463764    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"
Sep 06 05:19:12.463902 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.463783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq" event={"ID":"f3c9add5-e317-4e49-8702-b86e98da66c7","Type":"ContainerDied","Data":"3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384"}
Sep 06 05:19:12.463902 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.463815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq" event={"ID":"f3c9add5-e317-4e49-8702-b86e98da66c7","Type":"ContainerDied","Data":"707c321531cd4be6a92de7faa0a7db79006a73184280a54da0dc367d30dd7b4b"}
Sep 06 05:19:12.463902 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.463832    2580 scope.go:117] "RemoveContainer" containerID="3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384"
Sep 06 05:19:12.473331 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.473314    2580 scope.go:117] "RemoveContainer" containerID="3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384"
Sep 06 05:19:12.473616 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:19:12.473595    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384\": container with ID starting with 3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384 not found: ID does not exist" containerID="3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384"
Sep 06 05:19:12.473689 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.473628    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384"} err="failed to get container status \"3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384\": rpc error: code = NotFound desc = could not find container \"3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384\": container with ID starting with 3ab452a7237cf892b8af4dbf84e9c1ff30e2651132169cfebfb2c50727509384 not found: ID does not exist"
Sep 06 05:19:12.505252 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.505193    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"]
Sep 06 05:19:12.508053 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:12.508028    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1b19002b/e2e-lifecycle-model-1b19002b-kserve-59fb57b75-bbthq"]
Sep 06 05:19:14.348666 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:14.348634    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c9add5-e317-4e49-8702-b86e98da66c7" path="/var/lib/kubelet/pods/f3c9add5-e317-4e49-8702-b86e98da66c7/volumes"
Sep 06 05:19:19.337861 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:19.337769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:19:36.643466 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:36.643427    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:19:53.051705 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:19:53.051667    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:20:08.943368 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:08.943335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:20:10.694534 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.694503    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7552-df8297"]
Sep 06 05:20:10.694879 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.694870    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb447a8-1870-4566-8dbc-85f1175f2b4d" containerName="maas-api"
Sep 06 05:20:10.694921 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.694881    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb447a8-1870-4566-8dbc-85f1175f2b4d" containerName="maas-api"
Sep 06 05:20:10.694921 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.694898    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c9add5-e317-4e49-8702-b86e98da66c7" containerName="main"
Sep 06 05:20:10.694921 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.694905    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c9add5-e317-4e49-8702-b86e98da66c7" containerName="main"
Sep 06 05:20:10.695013 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.694980    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbb447a8-1870-4566-8dbc-85f1175f2b4d" containerName="maas-api"
Sep 06 05:20:10.695013 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.694991    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c9add5-e317-4e49-8702-b86e98da66c7" containerName="main"
Sep 06 05:20:10.698375 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.698348    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-df8297"
Sep 06 05:20:10.702093 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.702068    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7552-df8297"]
Sep 06 05:20:10.708689 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.708673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5l28p\""
Sep 06 05:20:10.821377 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.821343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cc9d\" (UniqueName: \"kubernetes.io/projected/2738081b-3e56-4227-9360-d8d610816a13-kube-api-access-5cc9d\") pod \"mt-curl-7552-df8297\" (UID: \"2738081b-3e56-4227-9360-d8d610816a13\") " pod="opendatahub/mt-curl-7552-df8297"
Sep 06 05:20:10.922367 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.922332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cc9d\" (UniqueName: \"kubernetes.io/projected/2738081b-3e56-4227-9360-d8d610816a13-kube-api-access-5cc9d\") pod \"mt-curl-7552-df8297\" (UID: \"2738081b-3e56-4227-9360-d8d610816a13\") " pod="opendatahub/mt-curl-7552-df8297"
Sep 06 05:20:10.937402 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:10.937367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cc9d\" (UniqueName: \"kubernetes.io/projected/2738081b-3e56-4227-9360-d8d610816a13-kube-api-access-5cc9d\") pod \"mt-curl-7552-df8297\" (UID: \"2738081b-3e56-4227-9360-d8d610816a13\") " pod="opendatahub/mt-curl-7552-df8297"
Sep 06 05:20:11.007914 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:11.007886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-df8297"
Sep 06 05:20:11.284044 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:20:11.284014    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2738081b_3e56_4227_9360_d8d610816a13.slice/crio-86ffa1ffdc3a05571cdcfea0574137b8a580259787f7c6e86440103ae0b44ac1 WatchSource:0}: Error finding container 86ffa1ffdc3a05571cdcfea0574137b8a580259787f7c6e86440103ae0b44ac1: Status 404 returned error can't find the container with id 86ffa1ffdc3a05571cdcfea0574137b8a580259787f7c6e86440103ae0b44ac1
Sep 06 05:20:11.285562 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:11.285538    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7552-df8297"]
Sep 06 05:20:11.798312 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:11.798271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7552-df8297" event={"ID":"2738081b-3e56-4227-9360-d8d610816a13","Type":"ContainerStarted","Data":"86ffa1ffdc3a05571cdcfea0574137b8a580259787f7c6e86440103ae0b44ac1"}
Sep 06 05:20:12.805898 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:12.805864    2580 generic.go:356] "Generic (PLEG): container finished" podID="2738081b-3e56-4227-9360-d8d610816a13" containerID="2ec94d3bcf6a91a86965ef52bca8f0c1262b3ac56d63e6ff7450789497cfa437" exitCode=0
Sep 06 05:20:12.806325 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:12.805917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7552-df8297" event={"ID":"2738081b-3e56-4227-9360-d8d610816a13","Type":"ContainerDied","Data":"2ec94d3bcf6a91a86965ef52bca8f0c1262b3ac56d63e6ff7450789497cfa437"}
Sep 06 05:20:13.946263 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:13.946241    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-df8297"
Sep 06 05:20:13.986948 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:13.986919    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7552-df8297_2738081b-3e56-4227-9360-d8d610816a13/mt-curl-7552-df8297/0.log"
Sep 06 05:20:14.011608 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:14.011570    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7552-df8297"]
Sep 06 05:20:14.016483 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:14.016461    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7552-df8297"]
Sep 06 05:20:14.050614 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:14.050590    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cc9d\" (UniqueName: \"kubernetes.io/projected/2738081b-3e56-4227-9360-d8d610816a13-kube-api-access-5cc9d\") pod \"2738081b-3e56-4227-9360-d8d610816a13\" (UID: \"2738081b-3e56-4227-9360-d8d610816a13\") "
Sep 06 05:20:14.052738 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:14.052719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2738081b-3e56-4227-9360-d8d610816a13-kube-api-access-5cc9d" (OuterVolumeSpecName: "kube-api-access-5cc9d") pod "2738081b-3e56-4227-9360-d8d610816a13" (UID: "2738081b-3e56-4227-9360-d8d610816a13"). InnerVolumeSpecName "kube-api-access-5cc9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:20:14.152140 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:14.152072    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cc9d\" (UniqueName: \"kubernetes.io/projected/2738081b-3e56-4227-9360-d8d610816a13-kube-api-access-5cc9d\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:20:14.347212 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:14.347179    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2738081b-3e56-4227-9360-d8d610816a13" path="/var/lib/kubelet/pods/2738081b-3e56-4227-9360-d8d610816a13/volumes"
Sep 06 05:20:14.818705 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:14.818676    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-df8297"
Sep 06 05:20:14.818886 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:14.818677    2580 scope.go:117] "RemoveContainer" containerID="2ec94d3bcf6a91a86965ef52bca8f0c1262b3ac56d63e6ff7450789497cfa437"
Sep 06 05:20:15.329499 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:15.329464    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"]
Sep 06 05:20:22.435949 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.435910    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"]
Sep 06 05:20:22.436356 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.436341    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2738081b-3e56-4227-9360-d8d610816a13" containerName="mt-curl-7552-df8297"
Sep 06 05:20:22.436356 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.436358    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2738081b-3e56-4227-9360-d8d610816a13" containerName="mt-curl-7552-df8297"
Sep 06 05:20:22.436452 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.436443    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2738081b-3e56-4227-9360-d8d610816a13" containerName="mt-curl-7552-df8297"
Sep 06 05:20:22.441106 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.441084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.443394 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.443367    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-494740\"/\"sub-test-model-494740-kserve-self-signed-certs\""
Sep 06 05:20:22.448361 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.448335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"]
Sep 06 05:20:22.527231 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.527208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-home\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.527343 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.527242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-model-cache\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.527343 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.527265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkmh\" (UniqueName: \"kubernetes.io/projected/7c63d897-bf0c-4086-ab08-c46d6c5802b4-kube-api-access-hhkmh\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.527435 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.527351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tmp-dir\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.527435 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.527404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tls-certs\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.527497 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.527439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-dshm\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.627918 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.627886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-home\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.628043 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.627927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-model-cache\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.628043 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.627962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkmh\" (UniqueName: \"kubernetes.io/projected/7c63d897-bf0c-4086-ab08-c46d6c5802b4-kube-api-access-hhkmh\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.628043 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.628011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tmp-dir\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.628248 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.628058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tls-certs\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.628248 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.628092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-dshm\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.628356 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.628284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-home\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.628412 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.628369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tmp-dir\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.628412 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.628369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-model-cache\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.630331 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.630313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-dshm\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.630578 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.630559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tls-certs\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.635438 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.635419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkmh\" (UniqueName: \"kubernetes.io/projected/7c63d897-bf0c-4086-ab08-c46d6c5802b4-kube-api-access-hhkmh\") pod \"sub-test-model-494740-kserve-64896bbcbc-p4c9l\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") " pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:22.754158 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:22.754130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:23.025730 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:23.025698    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"]
Sep 06 05:20:23.026667 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:20:23.026634    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c63d897_bf0c_4086_ab08_c46d6c5802b4.slice/crio-fbfb39dcb0026addb5ea4f76fc7743da4970dbf1ec56aac19ecce0e69bc54628 WatchSource:0}: Error finding container fbfb39dcb0026addb5ea4f76fc7743da4970dbf1ec56aac19ecce0e69bc54628: Status 404 returned error can't find the container with id fbfb39dcb0026addb5ea4f76fc7743da4970dbf1ec56aac19ecce0e69bc54628
Sep 06 05:20:23.874151 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:23.874118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l" event={"ID":"7c63d897-bf0c-4086-ab08-c46d6c5802b4","Type":"ContainerStarted","Data":"dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42"}
Sep 06 05:20:23.874151 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:23.874174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l" event={"ID":"7c63d897-bf0c-4086-ab08-c46d6c5802b4","Type":"ContainerStarted","Data":"fbfb39dcb0026addb5ea4f76fc7743da4970dbf1ec56aac19ecce0e69bc54628"}
Sep 06 05:20:23.890247 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:23.890196    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l" podStartSLOduration=1.890180518 podStartE2EDuration="1.890180518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:20:23.889386897 +0000 UTC m=+1500.030946851" watchObservedRunningTime="2026-09-06 05:20:23.890180518 +0000 UTC m=+1500.031740452"
Sep 06 05:20:24.478725 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:24.478691    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:20:24.481764 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:24.481743    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:20:27.650870 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:27.650838    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:20:30.351111 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.351064    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn" podUID="6d27d33b-eee8-4869-98d1-67e4e136ebf3" containerName="main" containerID="cri-o://f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282" gracePeriod=45
Sep 06 05:20:30.604227 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.604144    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:20:30.710737 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.710698    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-dshm\") pod \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") "
Sep 06 05:20:30.710935 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.710745    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tls-certs\") pod \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") "
Sep 06 05:20:30.710935 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.710804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-home\") pod \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") "
Sep 06 05:20:30.710935 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.710876    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tmp-dir\") pod \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") "
Sep 06 05:20:30.710935 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.710902    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-model-cache\") pod \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") "
Sep 06 05:20:30.710935 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.710930    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfx5x\" (UniqueName: \"kubernetes.io/projected/6d27d33b-eee8-4869-98d1-67e4e136ebf3-kube-api-access-zfx5x\") pod \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\" (UID: \"6d27d33b-eee8-4869-98d1-67e4e136ebf3\") "
Sep 06 05:20:30.711241 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.711186    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d27d33b-eee8-4869-98d1-67e4e136ebf3" (UID: "6d27d33b-eee8-4869-98d1-67e4e136ebf3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:20:30.711241 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.711207    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-home" (OuterVolumeSpecName: "home") pod "6d27d33b-eee8-4869-98d1-67e4e136ebf3" (UID: "6d27d33b-eee8-4869-98d1-67e4e136ebf3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:20:30.711341 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.711227    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d27d33b-eee8-4869-98d1-67e4e136ebf3" (UID: "6d27d33b-eee8-4869-98d1-67e4e136ebf3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:20:30.711341 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.711275    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-model-cache\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:20:30.711341 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.711293    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-home\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:20:30.713290 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.713267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-dshm" (OuterVolumeSpecName: "dshm") pod "6d27d33b-eee8-4869-98d1-67e4e136ebf3" (UID: "6d27d33b-eee8-4869-98d1-67e4e136ebf3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:20:30.713290 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.713267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d27d33b-eee8-4869-98d1-67e4e136ebf3" (UID: "6d27d33b-eee8-4869-98d1-67e4e136ebf3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:20:30.713436 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.713347    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d27d33b-eee8-4869-98d1-67e4e136ebf3-kube-api-access-zfx5x" (OuterVolumeSpecName: "kube-api-access-zfx5x") pod "6d27d33b-eee8-4869-98d1-67e4e136ebf3" (UID: "6d27d33b-eee8-4869-98d1-67e4e136ebf3"). InnerVolumeSpecName "kube-api-access-zfx5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:20:30.812214 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.812180    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tls-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:20:30.812214 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.812209    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-tmp-dir\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:20:30.812214 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.812218    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfx5x\" (UniqueName: \"kubernetes.io/projected/6d27d33b-eee8-4869-98d1-67e4e136ebf3-kube-api-access-zfx5x\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:20:30.812214 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.812226    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d27d33b-eee8-4869-98d1-67e4e136ebf3-dshm\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:20:30.918145 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.918059    2580 generic.go:356] "Generic (PLEG): container finished" podID="6d27d33b-eee8-4869-98d1-67e4e136ebf3" containerID="f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282" exitCode=0
Sep 06 05:20:30.918145 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.918109    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"
Sep 06 05:20:30.918145 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.918147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn" event={"ID":"6d27d33b-eee8-4869-98d1-67e4e136ebf3","Type":"ContainerDied","Data":"f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282"}
Sep 06 05:20:30.918423 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.918207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn" event={"ID":"6d27d33b-eee8-4869-98d1-67e4e136ebf3","Type":"ContainerDied","Data":"e20c18ca662656f8ea86ca7052cf0760b6025fa5ea1c1b7e377b659aebe1c6a5"}
Sep 06 05:20:30.918423 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.918230    2580 scope.go:117] "RemoveContainer" containerID="f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282"
Sep 06 05:20:30.928133 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.928116    2580 scope.go:117] "RemoveContainer" containerID="f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282"
Sep 06 05:20:30.928462 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:20:30.928443    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282\": container with ID starting with f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282 not found: ID does not exist" containerID="f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282"
Sep 06 05:20:30.928541 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.928472    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282"} err="failed to get container status \"f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282\": rpc error: code = NotFound desc = could not find container \"f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282\": container with ID starting with f23c550cdfb2f71cf947fb8cb702f0c3f69f8cd268d4ef034521f537d4991282 not found: ID does not exist"
Sep 06 05:20:30.939438 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.939402    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"]
Sep 06 05:20:30.942332 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:30.942307    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/auth-test-model-494740-kserve-79dd7db79f-27zcn"]
Sep 06 05:20:32.347315 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:32.347282    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d27d33b-eee8-4869-98d1-67e4e136ebf3" path="/var/lib/kubelet/pods/6d27d33b-eee8-4869-98d1-67e4e136ebf3/volumes"
Sep 06 05:20:32.754862 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:32.754830    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:32.755035 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:32.754880    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:32.767655 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:32.767631    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:32.941924 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:32.941897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:20:59.585059 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.585024    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"]
Sep 06 05:20:59.595016 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.594989    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d27d33b-eee8-4869-98d1-67e4e136ebf3" containerName="main"
Sep 06 05:20:59.595280 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.595265    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d27d33b-eee8-4869-98d1-67e4e136ebf3" containerName="main"
Sep 06 05:20:59.595599 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.595585    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d27d33b-eee8-4869-98d1-67e4e136ebf3" containerName="main"
Sep 06 05:20:59.601842 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.601819    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"]
Sep 06 05:20:59.601964 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.601907    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.603955 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.603931    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-71403e-serving-cert\""
Sep 06 05:20:59.656130 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.656105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-maas-api-tls\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.656250 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.656150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxh8q\" (UniqueName: \"kubernetes.io/projected/10b3b76c-8566-49a7-b76f-ede509c3598e-kube-api-access-sxh8q\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.656299 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.656250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-metrics-certs\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.656342 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.656307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b3b76c-8566-49a7-b76f-ede509c3598e-openshift-service-ca\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.757638 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.757617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-metrics-certs\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.757782 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.757662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b3b76c-8566-49a7-b76f-ede509c3598e-openshift-service-ca\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.757782 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.757713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-maas-api-tls\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.757782 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.757736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxh8q\" (UniqueName: \"kubernetes.io/projected/10b3b76c-8566-49a7-b76f-ede509c3598e-kube-api-access-sxh8q\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.757952 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:20:59.757862    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-serving-cert: secret "maas-api-e2e-api-a-71403e-serving-cert" not found
Sep 06 05:20:59.757952 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:20:59.757942    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-maas-api-tls podName:10b3b76c-8566-49a7-b76f-ede509c3598e nodeName:}" failed. No retries permitted until 2026-09-06 05:21:00.257922705 +0000 UTC m=+1536.399482631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-maas-api-tls") pod "maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n" (UID: "10b3b76c-8566-49a7-b76f-ede509c3598e") : secret "maas-api-e2e-api-a-71403e-serving-cert" not found
Sep 06 05:20:59.758399 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.758383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b3b76c-8566-49a7-b76f-ede509c3598e-openshift-service-ca\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.760314 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.760291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-metrics-certs\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:20:59.765516 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:20:59.765488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxh8q\" (UniqueName: \"kubernetes.io/projected/10b3b76c-8566-49a7-b76f-ede509c3598e-kube-api-access-sxh8q\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:21:00.262984 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:00.262954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-maas-api-tls\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:21:00.265434 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:00.265416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-maas-api-tls\") pod \"maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:21:00.512746 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:00.512716    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:21:00.785057 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:00.785023    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"]
Sep 06 05:21:00.786723 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:21:00.786697    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b3b76c_8566_49a7_b76f_ede509c3598e.slice/crio-7fc9590abc455ce2d8d90e606e4a51d10b65074c1401d97e9887228bad644470 WatchSource:0}: Error finding container 7fc9590abc455ce2d8d90e606e4a51d10b65074c1401d97e9887228bad644470: Status 404 returned error can't find the container with id 7fc9590abc455ce2d8d90e606e4a51d10b65074c1401d97e9887228bad644470
Sep 06 05:21:01.091837 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:01.091762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n" event={"ID":"10b3b76c-8566-49a7-b76f-ede509c3598e","Type":"ContainerStarted","Data":"7fc9590abc455ce2d8d90e606e4a51d10b65074c1401d97e9887228bad644470"}
Sep 06 05:21:02.099133 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:02.099098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n" event={"ID":"10b3b76c-8566-49a7-b76f-ede509c3598e","Type":"ContainerStarted","Data":"0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda"}
Sep 06 05:21:02.099618 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:02.099315    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:21:02.114548 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:02.114503    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n" podStartSLOduration=2.661006512 podStartE2EDuration="3.114463771s" totalImagesPullingTime="453.457259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:21:02.113503201 +0000 UTC m=+1538.255063134" watchObservedRunningTime="2026-09-06 05:21:02.114463771 +0000 UTC m=+1538.256023713"
Sep 06 05:21:08.111181 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:08.111129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:21:08.354337 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:08.354307    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:21:09.445061 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.445027    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"]
Sep 06 05:21:09.448867 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.448847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.452314 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.452293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-167b5b\"/\"default-dockercfg-v2ch9\""
Sep 06 05:21:09.452441 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.452293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-167b5b\"/\"sub-test-model-167b5b-kserve-self-signed-certs\""
Sep 06 05:21:09.452441 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.452362    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-167b5b\"/\"openshift-service-ca.crt\""
Sep 06 05:21:09.452441 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.452386    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-167b5b\"/\"kube-root-ca.crt\""
Sep 06 05:21:09.457324 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.457291    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"]
Sep 06 05:21:09.536634 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.536605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-dshm\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.536802 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.536675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hjtk\" (UniqueName: \"kubernetes.io/projected/81b15547-616e-4157-8070-22501c85eb62-kube-api-access-2hjtk\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.536802 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.536710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b15547-616e-4157-8070-22501c85eb62-tls-certs\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.536802 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.536740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-tmp-dir\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.536980 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.536816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-model-cache\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.536980 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.536857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-home\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.637832 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.637806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-dshm\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.637970 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.637855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hjtk\" (UniqueName: \"kubernetes.io/projected/81b15547-616e-4157-8070-22501c85eb62-kube-api-access-2hjtk\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.637970 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.637878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b15547-616e-4157-8070-22501c85eb62-tls-certs\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.637970 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.637903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-tmp-dir\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.637970 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.637953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-model-cache\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.638212 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.637977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-home\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.638335 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.638314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-tmp-dir\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.638402 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.638385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-model-cache\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.638488 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.638472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-home\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.640289 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.640270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-dshm\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.640529 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.640509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b15547-616e-4157-8070-22501c85eb62-tls-certs\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.645270 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.645249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hjtk\" (UniqueName: \"kubernetes.io/projected/81b15547-616e-4157-8070-22501c85eb62-kube-api-access-2hjtk\") pod \"sub-test-model-167b5b-kserve-5d687cd74f-8plrg\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:09.760509 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:09.760486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:10.033662 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:10.033636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"]
Sep 06 05:21:10.035428 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:21:10.035398    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81b15547_616e_4157_8070_22501c85eb62.slice/crio-a9e9bec10f00d6f71a267ebb6801f9b05e16b199f58866a6f5d1e368913034c5 WatchSource:0}: Error finding container a9e9bec10f00d6f71a267ebb6801f9b05e16b199f58866a6f5d1e368913034c5: Status 404 returned error can't find the container with id a9e9bec10f00d6f71a267ebb6801f9b05e16b199f58866a6f5d1e368913034c5
Sep 06 05:21:10.159302 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:10.159269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg" event={"ID":"81b15547-616e-4157-8070-22501c85eb62","Type":"ContainerStarted","Data":"7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2"}
Sep 06 05:21:10.159488 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:10.159315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg" event={"ID":"81b15547-616e-4157-8070-22501c85eb62","Type":"ContainerStarted","Data":"a9e9bec10f00d6f71a267ebb6801f9b05e16b199f58866a6f5d1e368913034c5"}
Sep 06 05:21:10.175576 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:10.175523    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg" podStartSLOduration=1.175505936 podStartE2EDuration="1.175505936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:21:10.173781291 +0000 UTC m=+1546.315341233" watchObservedRunningTime="2026-09-06 05:21:10.175505936 +0000 UTC m=+1546.317065879"
Sep 06 05:21:19.761542 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:19.761509    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:19.762010 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:19.761806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:19.774261 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:19.774239    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:20.226800 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:20.226775    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:21:22.146414 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:22.146380    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:21:24.543744 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:24.543713    2580 scope.go:117] "RemoveContainer" containerID="bdbd2f5e61ecc122820f4e9576bbd77147c1983a44870dac3fab8ed07bc84984"
Sep 06 05:21:41.151622 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:41.151587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:21:45.421428 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:45.421394    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7552-876c08"]
Sep 06 05:21:45.426377 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:45.426360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-876c08"
Sep 06 05:21:45.428582 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:45.428549    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5l28p\""
Sep 06 05:21:45.430514 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:45.430494    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7552-876c08"]
Sep 06 05:21:45.551886 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:45.551855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvc8w\" (UniqueName: \"kubernetes.io/projected/51402e4c-fac0-4de0-b957-9746d09c3c9e-kube-api-access-kvc8w\") pod \"mt-curl-7552-876c08\" (UID: \"51402e4c-fac0-4de0-b957-9746d09c3c9e\") " pod="opendatahub/mt-curl-7552-876c08"
Sep 06 05:21:45.652882 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:45.652845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvc8w\" (UniqueName: \"kubernetes.io/projected/51402e4c-fac0-4de0-b957-9746d09c3c9e-kube-api-access-kvc8w\") pod \"mt-curl-7552-876c08\" (UID: \"51402e4c-fac0-4de0-b957-9746d09c3c9e\") " pod="opendatahub/mt-curl-7552-876c08"
Sep 06 05:21:45.660201 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:45.660169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvc8w\" (UniqueName: \"kubernetes.io/projected/51402e4c-fac0-4de0-b957-9746d09c3c9e-kube-api-access-kvc8w\") pod \"mt-curl-7552-876c08\" (UID: \"51402e4c-fac0-4de0-b957-9746d09c3c9e\") " pod="opendatahub/mt-curl-7552-876c08"
Sep 06 05:21:45.737492 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:45.737461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-876c08"
Sep 06 05:21:46.005853 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:46.005831    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7552-876c08"]
Sep 06 05:21:46.008569 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:21:46.008537    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51402e4c_fac0_4de0_b957_9746d09c3c9e.slice/crio-1e9db5cf5c65fa69134997df82277eedc0b02764239869125d131ab5504219b3 WatchSource:0}: Error finding container 1e9db5cf5c65fa69134997df82277eedc0b02764239869125d131ab5504219b3: Status 404 returned error can't find the container with id 1e9db5cf5c65fa69134997df82277eedc0b02764239869125d131ab5504219b3
Sep 06 05:21:46.376727 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:46.376700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7552-876c08" event={"ID":"51402e4c-fac0-4de0-b957-9746d09c3c9e","Type":"ContainerStarted","Data":"25e53072c921b9323d8b69bf360482d8d656e282a655a76455c6b5d02eefb0ac"}
Sep 06 05:21:46.376858 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:46.376742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7552-876c08" event={"ID":"51402e4c-fac0-4de0-b957-9746d09c3c9e","Type":"ContainerStarted","Data":"1e9db5cf5c65fa69134997df82277eedc0b02764239869125d131ab5504219b3"}
Sep 06 05:21:46.391966 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:46.391919    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/mt-curl-7552-876c08" podStartSLOduration=1.117915599 podStartE2EDuration="1.391900865s" totalImagesPullingTime="273.985266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:21:46.389710889 +0000 UTC m=+1582.531270827" watchObservedRunningTime="2026-09-06 05:21:46.391900865 +0000 UTC m=+1582.533460808"
Sep 06 05:21:46.489099 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:46.489071    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7552-876c08_51402e4c-fac0-4de0-b957-9746d09c3c9e/mt-curl-7552-876c08/0.log"
Sep 06 05:21:47.384280 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:47.384246    2580 generic.go:356] "Generic (PLEG): container finished" podID="51402e4c-fac0-4de0-b957-9746d09c3c9e" containerID="25e53072c921b9323d8b69bf360482d8d656e282a655a76455c6b5d02eefb0ac" exitCode=0
Sep 06 05:21:47.384441 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:47.384321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7552-876c08" event={"ID":"51402e4c-fac0-4de0-b957-9746d09c3c9e","Type":"ContainerDied","Data":"25e53072c921b9323d8b69bf360482d8d656e282a655a76455c6b5d02eefb0ac"}
Sep 06 05:21:48.523384 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.523357    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-876c08"
Sep 06 05:21:48.572297 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.572271    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7552-876c08"]
Sep 06 05:21:48.579495 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.579474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvc8w\" (UniqueName: \"kubernetes.io/projected/51402e4c-fac0-4de0-b957-9746d09c3c9e-kube-api-access-kvc8w\") pod \"51402e4c-fac0-4de0-b957-9746d09c3c9e\" (UID: \"51402e4c-fac0-4de0-b957-9746d09c3c9e\") "
Sep 06 05:21:48.581609 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.581583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51402e4c-fac0-4de0-b957-9746d09c3c9e-kube-api-access-kvc8w" (OuterVolumeSpecName: "kube-api-access-kvc8w") pod "51402e4c-fac0-4de0-b957-9746d09c3c9e" (UID: "51402e4c-fac0-4de0-b957-9746d09c3c9e"). InnerVolumeSpecName "kube-api-access-kvc8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:21:48.595074 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.595054    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7552-876c08"]
Sep 06 05:21:48.680827 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.680772    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvc8w\" (UniqueName: \"kubernetes.io/projected/51402e4c-fac0-4de0-b957-9746d09c3c9e-kube-api-access-kvc8w\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:21:48.716936 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.716912    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7552-a3f0bd"]
Sep 06 05:21:48.717329 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.717317    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51402e4c-fac0-4de0-b957-9746d09c3c9e" containerName="mt-curl-7552-876c08"
Sep 06 05:21:48.717377 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.717332    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51402e4c-fac0-4de0-b957-9746d09c3c9e" containerName="mt-curl-7552-876c08"
Sep 06 05:21:48.717416 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.717400    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51402e4c-fac0-4de0-b957-9746d09c3c9e" containerName="mt-curl-7552-876c08"
Sep 06 05:21:48.720420 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.720406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-a3f0bd"
Sep 06 05:21:48.725715 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.725693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7552-a3f0bd"]
Sep 06 05:21:48.782014 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.781986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smt97\" (UniqueName: \"kubernetes.io/projected/c3f327eb-9766-434f-b330-1dd7c20ddc39-kube-api-access-smt97\") pod \"mt-curl-7552-a3f0bd\" (UID: \"c3f327eb-9766-434f-b330-1dd7c20ddc39\") " pod="opendatahub/mt-curl-7552-a3f0bd"
Sep 06 05:21:48.883279 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.883249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smt97\" (UniqueName: \"kubernetes.io/projected/c3f327eb-9766-434f-b330-1dd7c20ddc39-kube-api-access-smt97\") pod \"mt-curl-7552-a3f0bd\" (UID: \"c3f327eb-9766-434f-b330-1dd7c20ddc39\") " pod="opendatahub/mt-curl-7552-a3f0bd"
Sep 06 05:21:48.890254 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:48.890230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smt97\" (UniqueName: \"kubernetes.io/projected/c3f327eb-9766-434f-b330-1dd7c20ddc39-kube-api-access-smt97\") pod \"mt-curl-7552-a3f0bd\" (UID: \"c3f327eb-9766-434f-b330-1dd7c20ddc39\") " pod="opendatahub/mt-curl-7552-a3f0bd"
Sep 06 05:21:49.031234 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:49.031211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-a3f0bd"
Sep 06 05:21:49.305101 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:49.305073    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7552-a3f0bd"]
Sep 06 05:21:49.306587 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:21:49.306562    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3f327eb_9766_434f_b330_1dd7c20ddc39.slice/crio-4d03eeb2e5572c34c4ac895971fc03f57ac4fdcd07d87a2d1e1a498b9d517f9a WatchSource:0}: Error finding container 4d03eeb2e5572c34c4ac895971fc03f57ac4fdcd07d87a2d1e1a498b9d517f9a: Status 404 returned error can't find the container with id 4d03eeb2e5572c34c4ac895971fc03f57ac4fdcd07d87a2d1e1a498b9d517f9a
Sep 06 05:21:49.396711 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:49.396674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7552-a3f0bd" event={"ID":"c3f327eb-9766-434f-b330-1dd7c20ddc39","Type":"ContainerStarted","Data":"4d03eeb2e5572c34c4ac895971fc03f57ac4fdcd07d87a2d1e1a498b9d517f9a"}
Sep 06 05:21:49.397891 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:49.397872    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-876c08"
Sep 06 05:21:49.397891 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:49.397891    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e9db5cf5c65fa69134997df82277eedc0b02764239869125d131ab5504219b3"
Sep 06 05:21:49.407635 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:21:49.407608    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7552-876c08\" is forbidden: User \"system:node:ip-10-0-128-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-168.ec2.internal' and this object" podUID="51402e4c-fac0-4de0-b957-9746d09c3c9e" pod="opendatahub/mt-curl-7552-876c08"
Sep 06 05:21:50.347647 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:50.347611    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51402e4c-fac0-4de0-b957-9746d09c3c9e" path="/var/lib/kubelet/pods/51402e4c-fac0-4de0-b957-9746d09c3c9e/volumes"
Sep 06 05:21:50.404870 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:50.404839    2580 generic.go:356] "Generic (PLEG): container finished" podID="c3f327eb-9766-434f-b330-1dd7c20ddc39" containerID="af57f5afbcc82e041b721e0a44c5c23c104eab2dcb3c353175a6aa44a6f2b55f" exitCode=0
Sep 06 05:21:50.405003 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:50.404926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7552-a3f0bd" event={"ID":"c3f327eb-9766-434f-b330-1dd7c20ddc39","Type":"ContainerDied","Data":"af57f5afbcc82e041b721e0a44c5c23c104eab2dcb3c353175a6aa44a6f2b55f"}
Sep 06 05:21:51.554319 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:51.554297    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-a3f0bd"
Sep 06 05:21:51.572208 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:51.572189    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7552-a3f0bd_c3f327eb-9766-434f-b330-1dd7c20ddc39/mt-curl-7552-a3f0bd/0.log"
Sep 06 05:21:51.595355 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:51.595330    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7552-a3f0bd"]
Sep 06 05:21:51.597758 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:51.597737    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7552-a3f0bd"]
Sep 06 05:21:51.607828 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:51.607808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smt97\" (UniqueName: \"kubernetes.io/projected/c3f327eb-9766-434f-b330-1dd7c20ddc39-kube-api-access-smt97\") pod \"c3f327eb-9766-434f-b330-1dd7c20ddc39\" (UID: \"c3f327eb-9766-434f-b330-1dd7c20ddc39\") "
Sep 06 05:21:51.609965 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:51.609944    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f327eb-9766-434f-b330-1dd7c20ddc39-kube-api-access-smt97" (OuterVolumeSpecName: "kube-api-access-smt97") pod "c3f327eb-9766-434f-b330-1dd7c20ddc39" (UID: "c3f327eb-9766-434f-b330-1dd7c20ddc39"). InnerVolumeSpecName "kube-api-access-smt97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:21:51.709242 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:51.709189    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smt97\" (UniqueName: \"kubernetes.io/projected/c3f327eb-9766-434f-b330-1dd7c20ddc39-kube-api-access-smt97\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:21:52.349743 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:52.349711    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f327eb-9766-434f-b330-1dd7c20ddc39" path="/var/lib/kubelet/pods/c3f327eb-9766-434f-b330-1dd7c20ddc39/volumes"
Sep 06 05:21:52.418882 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:52.418859    2580 scope.go:117] "RemoveContainer" containerID="af57f5afbcc82e041b721e0a44c5c23c104eab2dcb3c353175a6aa44a6f2b55f"
Sep 06 05:21:52.418882 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:52.418887    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7552-a3f0bd"
Sep 06 05:21:53.542693 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:53.542654    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"]
Sep 06 05:21:54.553405 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:54.553373    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"]
Sep 06 05:21:57.440023 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:57.439983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:21:57.949169 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:57.949123    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"]
Sep 06 05:21:57.949590 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:57.949576    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f327eb-9766-434f-b330-1dd7c20ddc39" containerName="mt-curl-7552-a3f0bd"
Sep 06 05:21:57.949643 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:57.949592    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f327eb-9766-434f-b330-1dd7c20ddc39" containerName="mt-curl-7552-a3f0bd"
Sep 06 05:21:57.949689 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:57.949679    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f327eb-9766-434f-b330-1dd7c20ddc39" containerName="mt-curl-7552-a3f0bd"
Sep 06 05:21:57.952950 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:57.952935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:57.955540 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:57.955460    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-494740\"/\"rate-test-model-494740-kserve-self-signed-certs\""
Sep 06 05:21:57.961963 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:57.961943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"]
Sep 06 05:21:58.066810 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.066782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ktk\" (UniqueName: \"kubernetes.io/projected/d45c3479-ed16-462d-aa07-6009bef287a4-kube-api-access-q5ktk\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.066986 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.066831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-model-cache\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.066986 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.066853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-tmp-dir\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.066986 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.066939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-dshm\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.067126 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.066996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d45c3479-ed16-462d-aa07-6009bef287a4-tls-certs\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.067126 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.067017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-home\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.167841 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.167815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ktk\" (UniqueName: \"kubernetes.io/projected/d45c3479-ed16-462d-aa07-6009bef287a4-kube-api-access-q5ktk\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.167988 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.167854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-model-cache\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.167988 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.167871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-tmp-dir\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.167988 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.167896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-dshm\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.167988 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.167948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d45c3479-ed16-462d-aa07-6009bef287a4-tls-certs\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.167988 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.167980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-home\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.168268 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.168250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-model-cache\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.168340 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.168295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-tmp-dir\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.168340 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.168323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-home\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.170331 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.170296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-dshm\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.170572 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.170553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d45c3479-ed16-462d-aa07-6009bef287a4-tls-certs\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.175074 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.175047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ktk\" (UniqueName: \"kubernetes.io/projected/d45c3479-ed16-462d-aa07-6009bef287a4-kube-api-access-q5ktk\") pod \"rate-test-model-494740-kserve-7df47d7787-9rm2r\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") " pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.265163 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.265141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:21:58.537496 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:58.537469    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"]
Sep 06 05:21:58.539207 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:21:58.539149    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd45c3479_ed16_462d_aa07_6009bef287a4.slice/crio-0492354b5903e005de4bb89b8cdb16d6906f0bffb52fcf83ab573263d9a667bf WatchSource:0}: Error finding container 0492354b5903e005de4bb89b8cdb16d6906f0bffb52fcf83ab573263d9a667bf: Status 404 returned error can't find the container with id 0492354b5903e005de4bb89b8cdb16d6906f0bffb52fcf83ab573263d9a667bf
Sep 06 05:21:59.460956 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:59.460914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r" event={"ID":"d45c3479-ed16-462d-aa07-6009bef287a4","Type":"ContainerStarted","Data":"56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735"}
Sep 06 05:21:59.460956 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:59.460963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r" event={"ID":"d45c3479-ed16-462d-aa07-6009bef287a4","Type":"ContainerStarted","Data":"0492354b5903e005de4bb89b8cdb16d6906f0bffb52fcf83ab573263d9a667bf"}
Sep 06 05:21:59.479099 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:21:59.479043    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r" podStartSLOduration=2.47902827 podStartE2EDuration="2.47902827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:21:59.476702539 +0000 UTC m=+1595.618262481" watchObservedRunningTime="2026-09-06 05:21:59.47902827 +0000 UTC m=+1595.620588253"
Sep 06 05:22:08.265526 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.265496    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:22:08.265970 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.265560    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:22:08.277825 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.277802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:22:08.523559 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.523491    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:22:08.563507 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.563470    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l" podUID="7c63d897-bf0c-4086-ab08-c46d6c5802b4" containerName="main" containerID="cri-o://dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42" gracePeriod=45
Sep 06 05:22:08.817136 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.817113    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:22:08.970371 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970338    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-home\") pod \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") "
Sep 06 05:22:08.970535 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970390    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-model-cache\") pod \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") "
Sep 06 05:22:08.970535 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tls-certs\") pod \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") "
Sep 06 05:22:08.970535 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhkmh\" (UniqueName: \"kubernetes.io/projected/7c63d897-bf0c-4086-ab08-c46d6c5802b4-kube-api-access-hhkmh\") pod \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") "
Sep 06 05:22:08.970535 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970501    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-dshm\") pod \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") "
Sep 06 05:22:08.970535 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970523    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tmp-dir\") pod \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\" (UID: \"7c63d897-bf0c-4086-ab08-c46d6c5802b4\") "
Sep 06 05:22:08.970789 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970597    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-home" (OuterVolumeSpecName: "home") pod "7c63d897-bf0c-4086-ab08-c46d6c5802b4" (UID: "7c63d897-bf0c-4086-ab08-c46d6c5802b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:22:08.970789 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c63d897-bf0c-4086-ab08-c46d6c5802b4" (UID: "7c63d897-bf0c-4086-ab08-c46d6c5802b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:22:08.970901 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970783    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c63d897-bf0c-4086-ab08-c46d6c5802b4" (UID: "7c63d897-bf0c-4086-ab08-c46d6c5802b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:22:08.970901 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970886    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-home\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:08.971010 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970904    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-model-cache\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:08.971010 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.970918    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tmp-dir\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:08.972653 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.972624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-dshm" (OuterVolumeSpecName: "dshm") pod "7c63d897-bf0c-4086-ab08-c46d6c5802b4" (UID: "7c63d897-bf0c-4086-ab08-c46d6c5802b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:22:08.972778 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.972686    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c63d897-bf0c-4086-ab08-c46d6c5802b4" (UID: "7c63d897-bf0c-4086-ab08-c46d6c5802b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:22:08.972778 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:08.972727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c63d897-bf0c-4086-ab08-c46d6c5802b4-kube-api-access-hhkmh" (OuterVolumeSpecName: "kube-api-access-hhkmh") pod "7c63d897-bf0c-4086-ab08-c46d6c5802b4" (UID: "7c63d897-bf0c-4086-ab08-c46d6c5802b4"). InnerVolumeSpecName "kube-api-access-hhkmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:22:09.071436 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.071376    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c63d897-bf0c-4086-ab08-c46d6c5802b4-tls-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:09.071436 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.071405    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhkmh\" (UniqueName: \"kubernetes.io/projected/7c63d897-bf0c-4086-ab08-c46d6c5802b4-kube-api-access-hhkmh\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:09.071436 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.071415    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c63d897-bf0c-4086-ab08-c46d6c5802b4-dshm\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:09.519635 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.519600    2580 generic.go:356] "Generic (PLEG): container finished" podID="7c63d897-bf0c-4086-ab08-c46d6c5802b4" containerID="dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42" exitCode=0
Sep 06 05:22:09.520097 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.519648    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"
Sep 06 05:22:09.520097 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.519679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l" event={"ID":"7c63d897-bf0c-4086-ab08-c46d6c5802b4","Type":"ContainerDied","Data":"dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42"}
Sep 06 05:22:09.520097 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.519718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l" event={"ID":"7c63d897-bf0c-4086-ab08-c46d6c5802b4","Type":"ContainerDied","Data":"fbfb39dcb0026addb5ea4f76fc7743da4970dbf1ec56aac19ecce0e69bc54628"}
Sep 06 05:22:09.520097 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.519735    2580 scope.go:117] "RemoveContainer" containerID="dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42"
Sep 06 05:22:09.529800 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.529785    2580 scope.go:117] "RemoveContainer" containerID="dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42"
Sep 06 05:22:09.530056 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:22:09.530038    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42\": container with ID starting with dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42 not found: ID does not exist" containerID="dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42"
Sep 06 05:22:09.530099 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.530063    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42"} err="failed to get container status \"dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42\": rpc error: code = NotFound desc = could not find container \"dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42\": container with ID starting with dcc2c69dbbb9bed442e3741765fb00b22091eda497dbbf1ae0646c852e054a42 not found: ID does not exist"
Sep 06 05:22:09.540738 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.540700    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"]
Sep 06 05:22:09.543549 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.543527    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/sub-test-model-494740-kserve-64896bbcbc-p4c9l"]
Sep 06 05:22:09.574171 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.574109    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg" podUID="81b15547-616e-4157-8070-22501c85eb62" containerName="main" containerID="cri-o://7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2" gracePeriod=45
Sep 06 05:22:09.824415 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.824389    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:22:09.978314 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978279    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-tmp-dir\") pod \"81b15547-616e-4157-8070-22501c85eb62\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") "
Sep 06 05:22:09.978314 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-model-cache\") pod \"81b15547-616e-4157-8070-22501c85eb62\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") "
Sep 06 05:22:09.978508 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978392    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-dshm\") pod \"81b15547-616e-4157-8070-22501c85eb62\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") "
Sep 06 05:22:09.978508 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hjtk\" (UniqueName: \"kubernetes.io/projected/81b15547-616e-4157-8070-22501c85eb62-kube-api-access-2hjtk\") pod \"81b15547-616e-4157-8070-22501c85eb62\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") "
Sep 06 05:22:09.978508 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-home\") pod \"81b15547-616e-4157-8070-22501c85eb62\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") "
Sep 06 05:22:09.978508 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978490    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b15547-616e-4157-8070-22501c85eb62-tls-certs\") pod \"81b15547-616e-4157-8070-22501c85eb62\" (UID: \"81b15547-616e-4157-8070-22501c85eb62\") "
Sep 06 05:22:09.978708 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978600    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81b15547-616e-4157-8070-22501c85eb62" (UID: "81b15547-616e-4157-8070-22501c85eb62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:22:09.978708 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-model-cache" (OuterVolumeSpecName: "model-cache") pod "81b15547-616e-4157-8070-22501c85eb62" (UID: "81b15547-616e-4157-8070-22501c85eb62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:22:09.978806 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-home" (OuterVolumeSpecName: "home") pod "81b15547-616e-4157-8070-22501c85eb62" (UID: "81b15547-616e-4157-8070-22501c85eb62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:22:09.978806 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978799    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-tmp-dir\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:09.978913 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.978814    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-model-cache\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:09.980781 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.980750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-dshm" (OuterVolumeSpecName: "dshm") pod "81b15547-616e-4157-8070-22501c85eb62" (UID: "81b15547-616e-4157-8070-22501c85eb62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:22:09.980781 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.980765    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b15547-616e-4157-8070-22501c85eb62-kube-api-access-2hjtk" (OuterVolumeSpecName: "kube-api-access-2hjtk") pod "81b15547-616e-4157-8070-22501c85eb62" (UID: "81b15547-616e-4157-8070-22501c85eb62"). InnerVolumeSpecName "kube-api-access-2hjtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:22:09.980926 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:09.980802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b15547-616e-4157-8070-22501c85eb62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81b15547-616e-4157-8070-22501c85eb62" (UID: "81b15547-616e-4157-8070-22501c85eb62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:22:10.080041 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.079954    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-dshm\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:10.080041 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.079991    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hjtk\" (UniqueName: \"kubernetes.io/projected/81b15547-616e-4157-8070-22501c85eb62-kube-api-access-2hjtk\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:10.080041 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.080004    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b15547-616e-4157-8070-22501c85eb62-home\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:10.080041 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.080015    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b15547-616e-4157-8070-22501c85eb62-tls-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:10.347445 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.347366    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c63d897-bf0c-4086-ab08-c46d6c5802b4" path="/var/lib/kubelet/pods/7c63d897-bf0c-4086-ab08-c46d6c5802b4/volumes"
Sep 06 05:22:10.531044 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.531007    2580 generic.go:356] "Generic (PLEG): container finished" podID="81b15547-616e-4157-8070-22501c85eb62" containerID="7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2" exitCode=0
Sep 06 05:22:10.531538 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.531058    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"
Sep 06 05:22:10.531538 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.531085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg" event={"ID":"81b15547-616e-4157-8070-22501c85eb62","Type":"ContainerDied","Data":"7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2"}
Sep 06 05:22:10.531538 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.531127    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg" event={"ID":"81b15547-616e-4157-8070-22501c85eb62","Type":"ContainerDied","Data":"a9e9bec10f00d6f71a267ebb6801f9b05e16b199f58866a6f5d1e368913034c5"}
Sep 06 05:22:10.531538 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.531144    2580 scope.go:117] "RemoveContainer" containerID="7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2"
Sep 06 05:22:10.540585 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.540566    2580 scope.go:117] "RemoveContainer" containerID="7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2"
Sep 06 05:22:10.540845 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:22:10.540828    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2\": container with ID starting with 7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2 not found: ID does not exist" containerID="7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2"
Sep 06 05:22:10.540893 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.540853    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2"} err="failed to get container status \"7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2\": rpc error: code = NotFound desc = could not find container \"7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2\": container with ID starting with 7a3920131e63605978eccf56e0feba735bf17b5d83bf75657c0b39e785b472d2 not found: ID does not exist"
Sep 06 05:22:10.547534 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.547511    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"]
Sep 06 05:22:10.550372 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.550350    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/sub-test-model-167b5b-kserve-5d687cd74f-8plrg"]
Sep 06 05:22:10.917348 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.917320    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"]
Sep 06 05:22:10.917578 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:10.917547    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n" podUID="10b3b76c-8566-49a7-b76f-ede509c3598e" containerName="maas-api" containerID="cri-o://0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda" gracePeriod=30
Sep 06 05:22:11.164320 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.164289    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:22:11.291274 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.291244    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-metrics-certs\") pod \"10b3b76c-8566-49a7-b76f-ede509c3598e\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") "
Sep 06 05:22:11.291434 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.291339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b3b76c-8566-49a7-b76f-ede509c3598e-openshift-service-ca\") pod \"10b3b76c-8566-49a7-b76f-ede509c3598e\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") "
Sep 06 05:22:11.291434 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.291368    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxh8q\" (UniqueName: \"kubernetes.io/projected/10b3b76c-8566-49a7-b76f-ede509c3598e-kube-api-access-sxh8q\") pod \"10b3b76c-8566-49a7-b76f-ede509c3598e\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") "
Sep 06 05:22:11.291535 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.291432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-maas-api-tls\") pod \"10b3b76c-8566-49a7-b76f-ede509c3598e\" (UID: \"10b3b76c-8566-49a7-b76f-ede509c3598e\") "
Sep 06 05:22:11.291783 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.291754    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b3b76c-8566-49a7-b76f-ede509c3598e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10b3b76c-8566-49a7-b76f-ede509c3598e" (UID: "10b3b76c-8566-49a7-b76f-ede509c3598e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:22:11.293532 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.293510    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "10b3b76c-8566-49a7-b76f-ede509c3598e" (UID: "10b3b76c-8566-49a7-b76f-ede509c3598e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:22:11.293714 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.293694    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b3b76c-8566-49a7-b76f-ede509c3598e-kube-api-access-sxh8q" (OuterVolumeSpecName: "kube-api-access-sxh8q") pod "10b3b76c-8566-49a7-b76f-ede509c3598e" (UID: "10b3b76c-8566-49a7-b76f-ede509c3598e"). InnerVolumeSpecName "kube-api-access-sxh8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:22:11.293766 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.293715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10b3b76c-8566-49a7-b76f-ede509c3598e" (UID: "10b3b76c-8566-49a7-b76f-ede509c3598e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:22:11.392277 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.392249    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b3b76c-8566-49a7-b76f-ede509c3598e-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:11.392277 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.392273    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxh8q\" (UniqueName: \"kubernetes.io/projected/10b3b76c-8566-49a7-b76f-ede509c3598e-kube-api-access-sxh8q\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:11.392277 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.392283    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:11.392453 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.392292    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10b3b76c-8566-49a7-b76f-ede509c3598e-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:11.540657 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.540623    2580 generic.go:356] "Generic (PLEG): container finished" podID="10b3b76c-8566-49a7-b76f-ede509c3598e" containerID="0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda" exitCode=0
Sep 06 05:22:11.541035 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.540666    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"
Sep 06 05:22:11.541035 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.540683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n" event={"ID":"10b3b76c-8566-49a7-b76f-ede509c3598e","Type":"ContainerDied","Data":"0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda"}
Sep 06 05:22:11.541035 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.540733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n" event={"ID":"10b3b76c-8566-49a7-b76f-ede509c3598e","Type":"ContainerDied","Data":"7fc9590abc455ce2d8d90e606e4a51d10b65074c1401d97e9887228bad644470"}
Sep 06 05:22:11.541035 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.540765    2580 scope.go:117] "RemoveContainer" containerID="0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda"
Sep 06 05:22:11.551028 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.551009    2580 scope.go:117] "RemoveContainer" containerID="0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda"
Sep 06 05:22:11.551293 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:22:11.551273    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda\": container with ID starting with 0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda not found: ID does not exist" containerID="0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda"
Sep 06 05:22:11.551355 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.551305    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda"} err="failed to get container status \"0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda\": rpc error: code = NotFound desc = could not find container \"0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda\": container with ID starting with 0290b75dd5b869d8da006a5694da988d4802986408668270877435a46b1d5bda not found: ID does not exist"
Sep 06 05:22:11.561667 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.561645    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"]
Sep 06 05:22:11.563840 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:11.563821    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71403e-5bfb6fd54f-h2k6n"]
Sep 06 05:22:12.347714 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:12.347682    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10b3b76c-8566-49a7-b76f-ede509c3598e" path="/var/lib/kubelet/pods/10b3b76c-8566-49a7-b76f-ede509c3598e/volumes"
Sep 06 05:22:12.348057 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:12.348045    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b15547-616e-4157-8070-22501c85eb62" path="/var/lib/kubelet/pods/81b15547-616e-4157-8070-22501c85eb62/volumes"
Sep 06 05:22:15.040813 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:15.040776    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:22:19.637545 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.637512    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"]
Sep 06 05:22:19.638077 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.638057    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b15547-616e-4157-8070-22501c85eb62" containerName="main"
Sep 06 05:22:19.638077 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.638078    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b15547-616e-4157-8070-22501c85eb62" containerName="main"
Sep 06 05:22:19.638200 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.638090    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b3b76c-8566-49a7-b76f-ede509c3598e" containerName="maas-api"
Sep 06 05:22:19.638200 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.638097    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b3b76c-8566-49a7-b76f-ede509c3598e" containerName="maas-api"
Sep 06 05:22:19.638200 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.638125    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c63d897-bf0c-4086-ab08-c46d6c5802b4" containerName="main"
Sep 06 05:22:19.638200 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.638130    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c63d897-bf0c-4086-ab08-c46d6c5802b4" containerName="main"
Sep 06 05:22:19.638330 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.638215    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b15547-616e-4157-8070-22501c85eb62" containerName="main"
Sep 06 05:22:19.638330 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.638223    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="10b3b76c-8566-49a7-b76f-ede509c3598e" containerName="maas-api"
Sep 06 05:22:19.638330 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.638231    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c63d897-bf0c-4086-ab08-c46d6c5802b4" containerName="main"
Sep 06 05:22:19.642682 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.642664    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.646074 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.646053    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-167b5b\"/\"kube-root-ca.crt\""
Sep 06 05:22:19.646221 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.646054    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-167b5b\"/\"rate-test-model-167b5b-kserve-self-signed-certs\""
Sep 06 05:22:19.646221 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.646058    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-167b5b\"/\"openshift-service-ca.crt\""
Sep 06 05:22:19.646221 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.646062    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-167b5b\"/\"default-dockercfg-v2ch9\""
Sep 06 05:22:19.651679 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.651656    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"]
Sep 06 05:22:19.762533 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.762508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-model-cache\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.762674 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.762545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-home\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.762674 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.762582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-dshm\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.762674 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.762635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-tmp-dir\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.762674 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.762655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkbk\" (UniqueName: \"kubernetes.io/projected/feac0831-b7ed-4198-8d57-17f495b2b837-kube-api-access-zpkbk\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.762810 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.762680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feac0831-b7ed-4198-8d57-17f495b2b837-tls-certs\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.863261 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.863235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feac0831-b7ed-4198-8d57-17f495b2b837-tls-certs\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.863361 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.863306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-model-cache\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.863361 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.863341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-home\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.863468 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.863404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-dshm\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.863468 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.863436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-tmp-dir\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.863578 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.863462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkbk\" (UniqueName: \"kubernetes.io/projected/feac0831-b7ed-4198-8d57-17f495b2b837-kube-api-access-zpkbk\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.863793 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.863757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-model-cache\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.863878 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.863864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-home\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.863878 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.863872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-tmp-dir\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.865764 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.865736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-dshm\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.865994 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.865976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feac0831-b7ed-4198-8d57-17f495b2b837-tls-certs\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.870486 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.870466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkbk\" (UniqueName: \"kubernetes.io/projected/feac0831-b7ed-4198-8d57-17f495b2b837-kube-api-access-zpkbk\") pod \"rate-test-model-167b5b-kserve-67c68d96cf-khj9h\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") " pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:19.953493 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:19.953437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:20.228084 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:20.228057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"]
Sep 06 05:22:20.229901 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:22:20.229871    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeac0831_b7ed_4198_8d57_17f495b2b837.slice/crio-ede1f9f3b688df6b6339e25deaa6224913ce5f0e0e3f7ae6c4927d18d6500e13 WatchSource:0}: Error finding container ede1f9f3b688df6b6339e25deaa6224913ce5f0e0e3f7ae6c4927d18d6500e13: Status 404 returned error can't find the container with id ede1f9f3b688df6b6339e25deaa6224913ce5f0e0e3f7ae6c4927d18d6500e13
Sep 06 05:22:20.601644 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:20.601557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h" event={"ID":"feac0831-b7ed-4198-8d57-17f495b2b837","Type":"ContainerStarted","Data":"bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30"}
Sep 06 05:22:20.601644 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:20.601605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h" event={"ID":"feac0831-b7ed-4198-8d57-17f495b2b837","Type":"ContainerStarted","Data":"ede1f9f3b688df6b6339e25deaa6224913ce5f0e0e3f7ae6c4927d18d6500e13"}
Sep 06 05:22:20.616931 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:20.616864    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h" podStartSLOduration=1.616849287 podStartE2EDuration="1.616849287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:22:20.615876138 +0000 UTC m=+1616.757436099" watchObservedRunningTime="2026-09-06 05:22:20.616849287 +0000 UTC m=+1616.758409257"
Sep 06 05:22:29.954249 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:29.954211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:29.954249 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:29.954259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:29.966575 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:29.966547    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:30.679479 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:30.679448    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:22:34.743144 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:34.743113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:22:47.437744 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:47.437709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:22:50.530281 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.530248    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"]
Sep 06 05:22:50.530693 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.530476    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp" podUID="942b1faf-7d5e-4e0c-99a9-d05d493b2635" containerName="maas-api" containerID="cri-o://c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86" gracePeriod=30
Sep 06 05:22:50.778949 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.778924    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:22:50.781264 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.781204    2580 generic.go:356] "Generic (PLEG): container finished" podID="942b1faf-7d5e-4e0c-99a9-d05d493b2635" containerID="c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86" exitCode=0
Sep 06 05:22:50.781264 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.781244    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"
Sep 06 05:22:50.781389 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.781286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp" event={"ID":"942b1faf-7d5e-4e0c-99a9-d05d493b2635","Type":"ContainerDied","Data":"c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86"}
Sep 06 05:22:50.781389 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.781343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp" event={"ID":"942b1faf-7d5e-4e0c-99a9-d05d493b2635","Type":"ContainerDied","Data":"de96bcbd556aa29b7f5c7793bf34d76f67e8e6df98b5c3d8b7a778b309d14a72"}
Sep 06 05:22:50.781389 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.781365    2580 scope.go:117] "RemoveContainer" containerID="c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86"
Sep 06 05:22:50.790037 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.790012    2580 scope.go:117] "RemoveContainer" containerID="c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86"
Sep 06 05:22:50.790311 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:22:50.790296    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86\": container with ID starting with c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86 not found: ID does not exist" containerID="c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86"
Sep 06 05:22:50.790374 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.790319    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86"} err="failed to get container status \"c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86\": rpc error: code = NotFound desc = could not find container \"c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86\": container with ID starting with c475ba064c4f7b37c7087885d26a1c499a8d2eb77aafebdf620fcd7d5fe91c86 not found: ID does not exist"
Sep 06 05:22:50.837187 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.837165    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-metrics-certs\") pod \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") "
Sep 06 05:22:50.837379 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.837207    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns4qd\" (UniqueName: \"kubernetes.io/projected/942b1faf-7d5e-4e0c-99a9-d05d493b2635-kube-api-access-ns4qd\") pod \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") "
Sep 06 05:22:50.837379 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.837291    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-maas-api-tls\") pod \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") "
Sep 06 05:22:50.837508 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.837412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942b1faf-7d5e-4e0c-99a9-d05d493b2635-openshift-service-ca\") pod \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\" (UID: \"942b1faf-7d5e-4e0c-99a9-d05d493b2635\") "
Sep 06 05:22:50.837759 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.837736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942b1faf-7d5e-4e0c-99a9-d05d493b2635-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "942b1faf-7d5e-4e0c-99a9-d05d493b2635" (UID: "942b1faf-7d5e-4e0c-99a9-d05d493b2635"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:22:50.839288 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.839264    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "942b1faf-7d5e-4e0c-99a9-d05d493b2635" (UID: "942b1faf-7d5e-4e0c-99a9-d05d493b2635"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:22:50.839407 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.839386    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "942b1faf-7d5e-4e0c-99a9-d05d493b2635" (UID: "942b1faf-7d5e-4e0c-99a9-d05d493b2635"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:22:50.839458 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.839424    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942b1faf-7d5e-4e0c-99a9-d05d493b2635-kube-api-access-ns4qd" (OuterVolumeSpecName: "kube-api-access-ns4qd") pod "942b1faf-7d5e-4e0c-99a9-d05d493b2635" (UID: "942b1faf-7d5e-4e0c-99a9-d05d493b2635"). InnerVolumeSpecName "kube-api-access-ns4qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:22:50.938516 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.938477    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:50.938516 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.938505    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns4qd\" (UniqueName: \"kubernetes.io/projected/942b1faf-7d5e-4e0c-99a9-d05d493b2635-kube-api-access-ns4qd\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:50.938516 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.938515    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942b1faf-7d5e-4e0c-99a9-d05d493b2635-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:50.938516 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:50.938523    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942b1faf-7d5e-4e0c-99a9-d05d493b2635-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:22:51.101968 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:51.101940    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"]
Sep 06 05:22:51.107672 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:51.107651    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e24961-5c45f66447-vsqrp"]
Sep 06 05:22:52.347347 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:22:52.347309    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="942b1faf-7d5e-4e0c-99a9-d05d493b2635" path="/var/lib/kubelet/pods/942b1faf-7d5e-4e0c-99a9-d05d493b2635/volumes"
Sep 06 05:23:03.038301 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:03.038270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:23:14.444116 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:14.444081    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:23:21.075144 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:21.075112    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"]
Sep 06 05:23:22.051494 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:22.051458    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"]
Sep 06 05:23:24.611087 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:24.611052    2580 scope.go:117] "RemoveContainer" containerID="d754e1d7630862355225f4f5e6ccda4fe3d24fbf165eca437c3c4521ee679764"
Sep 06 05:23:24.619988 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:24.619968    2580 scope.go:117] "RemoveContainer" containerID="cdfb4f23a427e665dab6630b824f7470b763d5a0a6cf8845163a03fc0f6f556e"
Sep 06 05:23:25.238077 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:25.238038    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:23:36.095443 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.095388    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r" podUID="d45c3479-ed16-462d-aa07-6009bef287a4" containerName="main" containerID="cri-o://56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735" gracePeriod=45
Sep 06 05:23:36.346925 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.346865    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:23:36.433460 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433433    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-model-cache\") pod \"d45c3479-ed16-462d-aa07-6009bef287a4\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") "
Sep 06 05:23:36.433460 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433461    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-dshm\") pod \"d45c3479-ed16-462d-aa07-6009bef287a4\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") "
Sep 06 05:23:36.433651 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-home\") pod \"d45c3479-ed16-462d-aa07-6009bef287a4\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") "
Sep 06 05:23:36.433651 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433525    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-tmp-dir\") pod \"d45c3479-ed16-462d-aa07-6009bef287a4\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") "
Sep 06 05:23:36.433651 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433586    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5ktk\" (UniqueName: \"kubernetes.io/projected/d45c3479-ed16-462d-aa07-6009bef287a4-kube-api-access-q5ktk\") pod \"d45c3479-ed16-462d-aa07-6009bef287a4\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") "
Sep 06 05:23:36.433651 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433627    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d45c3479-ed16-462d-aa07-6009bef287a4-tls-certs\") pod \"d45c3479-ed16-462d-aa07-6009bef287a4\" (UID: \"d45c3479-ed16-462d-aa07-6009bef287a4\") "
Sep 06 05:23:36.433864 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "d45c3479-ed16-462d-aa07-6009bef287a4" (UID: "d45c3479-ed16-462d-aa07-6009bef287a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:23:36.433864 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-home" (OuterVolumeSpecName: "home") pod "d45c3479-ed16-462d-aa07-6009bef287a4" (UID: "d45c3479-ed16-462d-aa07-6009bef287a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:23:36.433864 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d45c3479-ed16-462d-aa07-6009bef287a4" (UID: "d45c3479-ed16-462d-aa07-6009bef287a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:23:36.434006 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433874    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-model-cache\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:36.434006 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433893    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-home\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:36.434006 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.433907    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-tmp-dir\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:36.435756 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.435732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-dshm" (OuterVolumeSpecName: "dshm") pod "d45c3479-ed16-462d-aa07-6009bef287a4" (UID: "d45c3479-ed16-462d-aa07-6009bef287a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:23:36.435756 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.435746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45c3479-ed16-462d-aa07-6009bef287a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d45c3479-ed16-462d-aa07-6009bef287a4" (UID: "d45c3479-ed16-462d-aa07-6009bef287a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:23:36.435920 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.435806    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45c3479-ed16-462d-aa07-6009bef287a4-kube-api-access-q5ktk" (OuterVolumeSpecName: "kube-api-access-q5ktk") pod "d45c3479-ed16-462d-aa07-6009bef287a4" (UID: "d45c3479-ed16-462d-aa07-6009bef287a4"). InnerVolumeSpecName "kube-api-access-q5ktk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:23:36.535081 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.535061    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d45c3479-ed16-462d-aa07-6009bef287a4-dshm\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:36.535081 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.535081    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5ktk\" (UniqueName: \"kubernetes.io/projected/d45c3479-ed16-462d-aa07-6009bef287a4-kube-api-access-q5ktk\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:36.535230 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:36.535091    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d45c3479-ed16-462d-aa07-6009bef287a4-tls-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:37.054463 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.054428    2580 generic.go:356] "Generic (PLEG): container finished" podID="d45c3479-ed16-462d-aa07-6009bef287a4" containerID="56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735" exitCode=0
Sep 06 05:23:37.054627 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.054487    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"
Sep 06 05:23:37.054627 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.054508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r" event={"ID":"d45c3479-ed16-462d-aa07-6009bef287a4","Type":"ContainerDied","Data":"56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735"}
Sep 06 05:23:37.054627 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.054555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r" event={"ID":"d45c3479-ed16-462d-aa07-6009bef287a4","Type":"ContainerDied","Data":"0492354b5903e005de4bb89b8cdb16d6906f0bffb52fcf83ab573263d9a667bf"}
Sep 06 05:23:37.054627 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.054572    2580 scope.go:117] "RemoveContainer" containerID="56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735"
Sep 06 05:23:37.063931 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.063914    2580 scope.go:117] "RemoveContainer" containerID="56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735"
Sep 06 05:23:37.064212 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:23:37.064194    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735\": container with ID starting with 56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735 not found: ID does not exist" containerID="56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735"
Sep 06 05:23:37.064273 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.064222    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735"} err="failed to get container status \"56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735\": rpc error: code = NotFound desc = could not find container \"56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735\": container with ID starting with 56ca5d12377f4f868e7827e53d2cb61a4848c4cfd44c4550934586f15e32a735 not found: ID does not exist"
Sep 06 05:23:37.072094 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.072066    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h" podUID="feac0831-b7ed-4198-8d57-17f495b2b837" containerName="main" containerID="cri-o://bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30" gracePeriod=45
Sep 06 05:23:37.075244 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.075222    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"]
Sep 06 05:23:37.078117 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.078097    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-494740/rate-test-model-494740-kserve-7df47d7787-9rm2r"]
Sep 06 05:23:37.313447 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.313385    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:23:37.442129 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.442104    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-model-cache\") pod \"feac0831-b7ed-4198-8d57-17f495b2b837\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") "
Sep 06 05:23:37.442286 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.442137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feac0831-b7ed-4198-8d57-17f495b2b837-tls-certs\") pod \"feac0831-b7ed-4198-8d57-17f495b2b837\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") "
Sep 06 05:23:37.442286 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.442217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-dshm\") pod \"feac0831-b7ed-4198-8d57-17f495b2b837\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") "
Sep 06 05:23:37.442286 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.442256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpkbk\" (UniqueName: \"kubernetes.io/projected/feac0831-b7ed-4198-8d57-17f495b2b837-kube-api-access-zpkbk\") pod \"feac0831-b7ed-4198-8d57-17f495b2b837\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") "
Sep 06 05:23:37.442286 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.442287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-home\") pod \"feac0831-b7ed-4198-8d57-17f495b2b837\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") "
Sep 06 05:23:37.442447 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.442306    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-tmp-dir\") pod \"feac0831-b7ed-4198-8d57-17f495b2b837\" (UID: \"feac0831-b7ed-4198-8d57-17f495b2b837\") "
Sep 06 05:23:37.442447 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.442383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-model-cache" (OuterVolumeSpecName: "model-cache") pod "feac0831-b7ed-4198-8d57-17f495b2b837" (UID: "feac0831-b7ed-4198-8d57-17f495b2b837"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:23:37.442645 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.442621    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "feac0831-b7ed-4198-8d57-17f495b2b837" (UID: "feac0831-b7ed-4198-8d57-17f495b2b837"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:23:37.442645 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.442638    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-model-cache\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:37.442950 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.442927    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-home" (OuterVolumeSpecName: "home") pod "feac0831-b7ed-4198-8d57-17f495b2b837" (UID: "feac0831-b7ed-4198-8d57-17f495b2b837"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:23:37.444545 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.444524    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feac0831-b7ed-4198-8d57-17f495b2b837-kube-api-access-zpkbk" (OuterVolumeSpecName: "kube-api-access-zpkbk") pod "feac0831-b7ed-4198-8d57-17f495b2b837" (UID: "feac0831-b7ed-4198-8d57-17f495b2b837"). InnerVolumeSpecName "kube-api-access-zpkbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:23:37.444545 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.444525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-dshm" (OuterVolumeSpecName: "dshm") pod "feac0831-b7ed-4198-8d57-17f495b2b837" (UID: "feac0831-b7ed-4198-8d57-17f495b2b837"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:23:37.444754 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.444738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feac0831-b7ed-4198-8d57-17f495b2b837-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "feac0831-b7ed-4198-8d57-17f495b2b837" (UID: "feac0831-b7ed-4198-8d57-17f495b2b837"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:23:37.543871 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.543844    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkbk\" (UniqueName: \"kubernetes.io/projected/feac0831-b7ed-4198-8d57-17f495b2b837-kube-api-access-zpkbk\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:37.543871 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.543867    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-home\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:37.544021 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.543877    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-tmp-dir\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:37.544021 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.543885    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feac0831-b7ed-4198-8d57-17f495b2b837-tls-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:37.544021 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.543891    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feac0831-b7ed-4198-8d57-17f495b2b837-dshm\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:23:37.751624 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:37.751592    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:23:38.061799 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.061704    2580 generic.go:356] "Generic (PLEG): container finished" podID="feac0831-b7ed-4198-8d57-17f495b2b837" containerID="bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30" exitCode=0
Sep 06 05:23:38.061799 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.061748    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"
Sep 06 05:23:38.061799 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.061790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h" event={"ID":"feac0831-b7ed-4198-8d57-17f495b2b837","Type":"ContainerDied","Data":"bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30"}
Sep 06 05:23:38.062085 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.061839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h" event={"ID":"feac0831-b7ed-4198-8d57-17f495b2b837","Type":"ContainerDied","Data":"ede1f9f3b688df6b6339e25deaa6224913ce5f0e0e3f7ae6c4927d18d6500e13"}
Sep 06 05:23:38.062085 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.061860    2580 scope.go:117] "RemoveContainer" containerID="bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30"
Sep 06 05:23:38.071376 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.071358    2580 scope.go:117] "RemoveContainer" containerID="bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30"
Sep 06 05:23:38.071652 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:23:38.071635    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30\": container with ID starting with bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30 not found: ID does not exist" containerID="bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30"
Sep 06 05:23:38.071704 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.071669    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30"} err="failed to get container status \"bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30\": rpc error: code = NotFound desc = could not find container \"bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30\": container with ID starting with bbaf46b9694b0dee9d0cb49d702cef462fcb6a8f2248a383e63112ef276c5c30 not found: ID does not exist"
Sep 06 05:23:38.081549 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.081528    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"]
Sep 06 05:23:38.083652 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.083635    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-167b5b/rate-test-model-167b5b-kserve-67c68d96cf-khj9h"]
Sep 06 05:23:38.347283 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.347222    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d45c3479-ed16-462d-aa07-6009bef287a4" path="/var/lib/kubelet/pods/d45c3479-ed16-462d-aa07-6009bef287a4/volumes"
Sep 06 05:23:38.347615 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:38.347601    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feac0831-b7ed-4198-8d57-17f495b2b837" path="/var/lib/kubelet/pods/feac0831-b7ed-4198-8d57-17f495b2b837/volumes"
Sep 06 05:23:55.593256 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.593221    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"]
Sep 06 05:23:55.593868 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.593846    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942b1faf-7d5e-4e0c-99a9-d05d493b2635" containerName="maas-api"
Sep 06 05:23:55.593942 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.593872    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="942b1faf-7d5e-4e0c-99a9-d05d493b2635" containerName="maas-api"
Sep 06 05:23:55.593942 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.593884    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d45c3479-ed16-462d-aa07-6009bef287a4" containerName="main"
Sep 06 05:23:55.593942 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.593893    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d45c3479-ed16-462d-aa07-6009bef287a4" containerName="main"
Sep 06 05:23:55.593942 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.593928    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feac0831-b7ed-4198-8d57-17f495b2b837" containerName="main"
Sep 06 05:23:55.593942 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.593937    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="feac0831-b7ed-4198-8d57-17f495b2b837" containerName="main"
Sep 06 05:23:55.594262 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.594029    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="942b1faf-7d5e-4e0c-99a9-d05d493b2635" containerName="maas-api"
Sep 06 05:23:55.594262 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.594042    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="feac0831-b7ed-4198-8d57-17f495b2b837" containerName="main"
Sep 06 05:23:55.594262 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.594056    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d45c3479-ed16-462d-aa07-6009bef287a4" containerName="main"
Sep 06 05:23:55.597792 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.597773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.600269 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.600035    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-22321a-serving-cert\""
Sep 06 05:23:55.604106 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.604038    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"]
Sep 06 05:23:55.697306 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.697276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344851fe-2fdc-4225-8f15-2d294389e152-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.697412 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.697313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-metrics-certs\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.697412 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.697337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26mr5\" (UniqueName: \"kubernetes.io/projected/344851fe-2fdc-4225-8f15-2d294389e152-kube-api-access-26mr5\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.697412 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.697383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-maas-api-tls\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.798488 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.798455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-maas-api-tls\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.798625 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.798565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344851fe-2fdc-4225-8f15-2d294389e152-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.798625 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.798601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-metrics-certs\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.798749 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.798632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26mr5\" (UniqueName: \"kubernetes.io/projected/344851fe-2fdc-4225-8f15-2d294389e152-kube-api-access-26mr5\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.799341 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.799269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344851fe-2fdc-4225-8f15-2d294389e152-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.800965 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.800939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-maas-api-tls\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.801050 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.801039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-metrics-certs\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.805874 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.805855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26mr5\" (UniqueName: \"kubernetes.io/projected/344851fe-2fdc-4225-8f15-2d294389e152-kube-api-access-26mr5\") pod \"maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:55.912213 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:55.912131    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:56.187711 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:56.187687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"]
Sep 06 05:23:56.189549 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:23:56.189519    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344851fe_2fdc_4225_8f15_2d294389e152.slice/crio-f56d59c20c8bee88f1939452a4dbd24ae736c06a588e49b8e628f2026b5c5277 WatchSource:0}: Error finding container f56d59c20c8bee88f1939452a4dbd24ae736c06a588e49b8e628f2026b5c5277: Status 404 returned error can't find the container with id f56d59c20c8bee88f1939452a4dbd24ae736c06a588e49b8e628f2026b5c5277
Sep 06 05:23:56.190888 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:56.190868    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 05:23:57.175308 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:57.175266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg" event={"ID":"344851fe-2fdc-4225-8f15-2d294389e152","Type":"ContainerStarted","Data":"df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd"}
Sep 06 05:23:57.175731 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:57.175326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg" event={"ID":"344851fe-2fdc-4225-8f15-2d294389e152","Type":"ContainerStarted","Data":"f56d59c20c8bee88f1939452a4dbd24ae736c06a588e49b8e628f2026b5c5277"}
Sep 06 05:23:57.175731 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:57.175366    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:23:57.192557 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:23:57.192509    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg" podStartSLOduration=1.8207051779999999 podStartE2EDuration="2.192493328s" totalImagesPullingTime="371.78815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:23:57.19064139 +0000 UTC m=+1713.332201350" watchObservedRunningTime="2026-09-06 05:23:57.192493328 +0000 UTC m=+1713.334053302"
Sep 06 05:24:02.651544 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:02.651504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:24:03.186831 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:03.186802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:24:42.147372 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.147337    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"]
Sep 06 05:24:42.151110 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.151089    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.153300 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.153265    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-41fda0\"/\"openshift-service-ca.crt\""
Sep 06 05:24:42.153413 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.153369    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-41fda0\"/\"ipp-route-41fda0-kserve-self-signed-certs\""
Sep 06 05:24:42.154256 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.154233    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-41fda0\"/\"kube-root-ca.crt\""
Sep 06 05:24:42.154256 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.154251    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-41fda0\"/\"default-dockercfg-d8w9c\""
Sep 06 05:24:42.163364 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.163341    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"]
Sep 06 05:24:42.294379 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.294346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-tmp-dir\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.294547 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.294398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-model-cache\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.294547 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.294428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgzpk\" (UniqueName: \"kubernetes.io/projected/51e2be20-6340-482f-89e4-5379bb84b016-kube-api-access-rgzpk\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.294547 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.294498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-home\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.294662 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.294565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e2be20-6340-482f-89e4-5379bb84b016-tls-certs\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.294662 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.294585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-dshm\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.395319 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.395292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-tmp-dir\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.395475 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.395336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-model-cache\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.395475 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.395356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgzpk\" (UniqueName: \"kubernetes.io/projected/51e2be20-6340-482f-89e4-5379bb84b016-kube-api-access-rgzpk\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.395475 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.395378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-home\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.395475 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.395424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e2be20-6340-482f-89e4-5379bb84b016-tls-certs\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.395475 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.395451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-dshm\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.395756 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.395735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-model-cache\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.395805 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.395753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-tmp-dir\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.395842 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.395801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-home\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.397898 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.397836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-dshm\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.398086 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.398069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e2be20-6340-482f-89e4-5379bb84b016-tls-certs\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.402412 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.402389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgzpk\" (UniqueName: \"kubernetes.io/projected/51e2be20-6340-482f-89e4-5379bb84b016-kube-api-access-rgzpk\") pod \"ipp-route-41fda0-kserve-555cf55b67-7ll99\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") " pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.461516 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.461484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:42.734895 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:42.734870    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"]
Sep 06 05:24:42.736259 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:24:42.736234    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51e2be20_6340_482f_89e4_5379bb84b016.slice/crio-dc5a35eee7ca51dee300edcba808b68645b8aab128a92963bbecfccc4a9986c6 WatchSource:0}: Error finding container dc5a35eee7ca51dee300edcba808b68645b8aab128a92963bbecfccc4a9986c6: Status 404 returned error can't find the container with id dc5a35eee7ca51dee300edcba808b68645b8aab128a92963bbecfccc4a9986c6
Sep 06 05:24:43.431662 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:43.431624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99" event={"ID":"51e2be20-6340-482f-89e4-5379bb84b016","Type":"ContainerStarted","Data":"272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766"}
Sep 06 05:24:43.431662 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:43.431663    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99" event={"ID":"51e2be20-6340-482f-89e4-5379bb84b016","Type":"ContainerStarted","Data":"dc5a35eee7ca51dee300edcba808b68645b8aab128a92963bbecfccc4a9986c6"}
Sep 06 05:24:43.448435 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:43.448381    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99" podStartSLOduration=1.448365972 podStartE2EDuration="1.448365972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:24:43.446583047 +0000 UTC m=+1759.588142989" watchObservedRunningTime="2026-09-06 05:24:43.448365972 +0000 UTC m=+1759.589925915"
Sep 06 05:24:45.654183 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:45.649674    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:24:52.461906 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:52.461874    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:52.462374 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:52.461929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:52.474507 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:52.474487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:24:52.491695 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:24:52.491669    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:25:01.445755 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:01.445713    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:25:13.052567 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.052528    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"]
Sep 06 05:25:13.056344 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.056321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.058647 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.058625    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-c0aaed-serving-cert\""
Sep 06 05:25:13.064615 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.064584    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"]
Sep 06 05:25:13.145699 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.145676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.145823 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.145703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsgqh\" (UniqueName: \"kubernetes.io/projected/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-kube-api-access-qsgqh\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.145823 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.145765    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.145823 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.145808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.247012 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.246985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.247141 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.247017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsgqh\" (UniqueName: \"kubernetes.io/projected/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-kube-api-access-qsgqh\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.247141 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.247061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.247141 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.247108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.247332 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:25:13.247168    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-serving-cert: secret "maas-api-e2e-ipp-cleanup-c0aaed-serving-cert" not found
Sep 06 05:25:13.247332 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:25:13.247246    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-maas-api-tls podName:d19c6cde-4aa7-4c42-8d0c-cd7106b66b71 nodeName:}" failed. No retries permitted until 2026-09-06 05:25:13.7472241 +0000 UTC m=+1789.888784027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn" (UID: "d19c6cde-4aa7-4c42-8d0c-cd7106b66b71") : secret "maas-api-e2e-ipp-cleanup-c0aaed-serving-cert" not found
Sep 06 05:25:13.247774 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.247751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.249680 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.249660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.256094 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.256074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsgqh\" (UniqueName: \"kubernetes.io/projected/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-kube-api-access-qsgqh\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.751392 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.751356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.753898 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.753874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:13.971278 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:13.971244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:14.248268 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:25:14.248224    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19c6cde_4aa7_4c42_8d0c_cd7106b66b71.slice/crio-aadaba2b7b6f6e59f9e2cdd5a604567fe6ca7cedc6bd4e4467686d4b821d7cb4 WatchSource:0}: Error finding container aadaba2b7b6f6e59f9e2cdd5a604567fe6ca7cedc6bd4e4467686d4b821d7cb4: Status 404 returned error can't find the container with id aadaba2b7b6f6e59f9e2cdd5a604567fe6ca7cedc6bd4e4467686d4b821d7cb4
Sep 06 05:25:14.250621 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:14.250599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"]
Sep 06 05:25:14.605419 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:14.605385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn" event={"ID":"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71","Type":"ContainerStarted","Data":"aadaba2b7b6f6e59f9e2cdd5a604567fe6ca7cedc6bd4e4467686d4b821d7cb4"}
Sep 06 05:25:15.612663 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:15.612577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn" event={"ID":"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71","Type":"ContainerStarted","Data":"e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4"}
Sep 06 05:25:15.613023 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:15.612664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:15.630604 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:15.630553    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn" podStartSLOduration=2.256235693 podStartE2EDuration="2.630539635s" totalImagesPullingTime="374.303942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:25:15.628637957 +0000 UTC m=+1791.770197916" watchObservedRunningTime="2026-09-06 05:25:15.630539635 +0000 UTC m=+1791.772099571"
Sep 06 05:25:21.147896 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:21.147860    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:25:21.623946 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:21.623917    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:24.518268 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:24.518240    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:25:24.522536 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:24.522517    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:25:43.375302 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.375267    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"]
Sep 06 05:25:43.375738 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.375552    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn" podUID="d19c6cde-4aa7-4c42-8d0c-cd7106b66b71" containerName="maas-api" containerID="cri-o://e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4" gracePeriod=30
Sep 06 05:25:43.624316 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.624296    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:43.631206 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.631137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-openshift-service-ca\") pod \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") "
Sep 06 05:25:43.631206 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.631195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsgqh\" (UniqueName: \"kubernetes.io/projected/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-kube-api-access-qsgqh\") pod \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") "
Sep 06 05:25:43.631385 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.631235    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-maas-api-tls\") pod \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") "
Sep 06 05:25:43.631385 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.631259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-metrics-certs\") pod \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\" (UID: \"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71\") "
Sep 06 05:25:43.631554 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.631532    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d19c6cde-4aa7-4c42-8d0c-cd7106b66b71" (UID: "d19c6cde-4aa7-4c42-8d0c-cd7106b66b71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:25:43.633368 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.633341    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d19c6cde-4aa7-4c42-8d0c-cd7106b66b71" (UID: "d19c6cde-4aa7-4c42-8d0c-cd7106b66b71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:25:43.633465 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.633448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-kube-api-access-qsgqh" (OuterVolumeSpecName: "kube-api-access-qsgqh") pod "d19c6cde-4aa7-4c42-8d0c-cd7106b66b71" (UID: "d19c6cde-4aa7-4c42-8d0c-cd7106b66b71"). InnerVolumeSpecName "kube-api-access-qsgqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:25:43.633465 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.633448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d19c6cde-4aa7-4c42-8d0c-cd7106b66b71" (UID: "d19c6cde-4aa7-4c42-8d0c-cd7106b66b71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:25:43.732582 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.732547    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:25:43.732582 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.732586    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsgqh\" (UniqueName: \"kubernetes.io/projected/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-kube-api-access-qsgqh\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:25:43.732754 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.732602    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:25:43.732754 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.732615    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:25:43.771697 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.771669    2580 generic.go:356] "Generic (PLEG): container finished" podID="d19c6cde-4aa7-4c42-8d0c-cd7106b66b71" containerID="e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4" exitCode=0
Sep 06 05:25:43.771831 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.771712    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"
Sep 06 05:25:43.771831 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.771751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn" event={"ID":"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71","Type":"ContainerDied","Data":"e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4"}
Sep 06 05:25:43.771831 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.771800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn" event={"ID":"d19c6cde-4aa7-4c42-8d0c-cd7106b66b71","Type":"ContainerDied","Data":"aadaba2b7b6f6e59f9e2cdd5a604567fe6ca7cedc6bd4e4467686d4b821d7cb4"}
Sep 06 05:25:43.771831 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.771822    2580 scope.go:117] "RemoveContainer" containerID="e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4"
Sep 06 05:25:43.782234 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.782210    2580 scope.go:117] "RemoveContainer" containerID="e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4"
Sep 06 05:25:43.784834 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:25:43.784804    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4\": container with ID starting with e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4 not found: ID does not exist" containerID="e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4"
Sep 06 05:25:43.784937 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.784849    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4"} err="failed to get container status \"e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4\": rpc error: code = NotFound desc = could not find container \"e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4\": container with ID starting with e04e32dcd70dd42a7439604ba2d5877f684ee3cf49edf4ae5288d7ccdd1175f4 not found: ID does not exist"
Sep 06 05:25:43.792843 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.792818    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"]
Sep 06 05:25:43.796344 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:43.796322    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c0aaed-8555d4949-5n5wn"]
Sep 06 05:25:44.348762 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:44.348734    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d19c6cde-4aa7-4c42-8d0c-cd7106b66b71" path="/var/lib/kubelet/pods/d19c6cde-4aa7-4c42-8d0c-cd7106b66b71/volumes"
Sep 06 05:25:48.442247 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:25:48.442217    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:26:09.643630 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:09.643589    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:26:14.477818 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:14.477787    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"]
Sep 06 05:26:19.044471 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:19.044429    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:26:29.498693 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.498654    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99" podUID="51e2be20-6340-482f-89e4-5379bb84b016" containerName="main" containerID="cri-o://272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766" gracePeriod=45
Sep 06 05:26:29.740803 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.740779    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:26:29.920353 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920274    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-dshm\") pod \"51e2be20-6340-482f-89e4-5379bb84b016\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") "
Sep 06 05:26:29.920353 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920304    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-tmp-dir\") pod \"51e2be20-6340-482f-89e4-5379bb84b016\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") "
Sep 06 05:26:29.920353 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920331    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-model-cache\") pod \"51e2be20-6340-482f-89e4-5379bb84b016\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") "
Sep 06 05:26:29.920353 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-home\") pod \"51e2be20-6340-482f-89e4-5379bb84b016\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") "
Sep 06 05:26:29.920682 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920384    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgzpk\" (UniqueName: \"kubernetes.io/projected/51e2be20-6340-482f-89e4-5379bb84b016-kube-api-access-rgzpk\") pod \"51e2be20-6340-482f-89e4-5379bb84b016\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") "
Sep 06 05:26:29.920682 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e2be20-6340-482f-89e4-5379bb84b016-tls-certs\") pod \"51e2be20-6340-482f-89e4-5379bb84b016\" (UID: \"51e2be20-6340-482f-89e4-5379bb84b016\") "
Sep 06 05:26:29.920682 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920566    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51e2be20-6340-482f-89e4-5379bb84b016" (UID: "51e2be20-6340-482f-89e4-5379bb84b016"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:26:29.920682 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920596    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-model-cache" (OuterVolumeSpecName: "model-cache") pod "51e2be20-6340-482f-89e4-5379bb84b016" (UID: "51e2be20-6340-482f-89e4-5379bb84b016"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:26:29.920682 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920636    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-home" (OuterVolumeSpecName: "home") pod "51e2be20-6340-482f-89e4-5379bb84b016" (UID: "51e2be20-6340-482f-89e4-5379bb84b016"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:26:29.920895 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920746    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-model-cache\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:26:29.920895 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920761    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-home\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:26:29.920895 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.920774    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-tmp-dir\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:26:29.922679 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.922653    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e2be20-6340-482f-89e4-5379bb84b016-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51e2be20-6340-482f-89e4-5379bb84b016" (UID: "51e2be20-6340-482f-89e4-5379bb84b016"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:26:29.922785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.922697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-dshm" (OuterVolumeSpecName: "dshm") pod "51e2be20-6340-482f-89e4-5379bb84b016" (UID: "51e2be20-6340-482f-89e4-5379bb84b016"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 05:26:29.922785 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:29.922715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e2be20-6340-482f-89e4-5379bb84b016-kube-api-access-rgzpk" (OuterVolumeSpecName: "kube-api-access-rgzpk") pod "51e2be20-6340-482f-89e4-5379bb84b016" (UID: "51e2be20-6340-482f-89e4-5379bb84b016"). InnerVolumeSpecName "kube-api-access-rgzpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:26:30.021283 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.021257    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e2be20-6340-482f-89e4-5379bb84b016-dshm\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:26:30.021283 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.021281    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgzpk\" (UniqueName: \"kubernetes.io/projected/51e2be20-6340-482f-89e4-5379bb84b016-kube-api-access-rgzpk\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:26:30.021421 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.021292    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e2be20-6340-482f-89e4-5379bb84b016-tls-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:26:30.035390 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.035363    2580 generic.go:356] "Generic (PLEG): container finished" podID="51e2be20-6340-482f-89e4-5379bb84b016" containerID="272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766" exitCode=0
Sep 06 05:26:30.035490 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.035420    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"
Sep 06 05:26:30.035490 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.035447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99" event={"ID":"51e2be20-6340-482f-89e4-5379bb84b016","Type":"ContainerDied","Data":"272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766"}
Sep 06 05:26:30.035562 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.035490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99" event={"ID":"51e2be20-6340-482f-89e4-5379bb84b016","Type":"ContainerDied","Data":"dc5a35eee7ca51dee300edcba808b68645b8aab128a92963bbecfccc4a9986c6"}
Sep 06 05:26:30.035562 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.035511    2580 scope.go:117] "RemoveContainer" containerID="272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766"
Sep 06 05:26:30.044876 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.044859    2580 scope.go:117] "RemoveContainer" containerID="272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766"
Sep 06 05:26:30.045182 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:26:30.045137    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766\": container with ID starting with 272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766 not found: ID does not exist" containerID="272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766"
Sep 06 05:26:30.045273 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.045185    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766"} err="failed to get container status \"272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766\": rpc error: code = NotFound desc = could not find container \"272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766\": container with ID starting with 272658aacf4f006b7df3378a6a1be91b7b113e5ab865c523fd29d204b944d766 not found: ID does not exist"
Sep 06 05:26:30.055076 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.055055    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"]
Sep 06 05:26:30.058332 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.058312    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-41fda0/ipp-route-41fda0-kserve-555cf55b67-7ll99"]
Sep 06 05:26:30.347670 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:26:30.347641    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e2be20-6340-482f-89e4-5379bb84b016" path="/var/lib/kubelet/pods/51e2be20-6340-482f-89e4-5379bb84b016/volumes"
Sep 06 05:27:08.201825 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.201792    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"]
Sep 06 05:27:08.202298 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.202001    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg" podUID="344851fe-2fdc-4225-8f15-2d294389e152" containerName="maas-api" containerID="cri-o://df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd" gracePeriod=30
Sep 06 05:27:08.453096 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.453041    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:27:08.560405 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.560378    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26mr5\" (UniqueName: \"kubernetes.io/projected/344851fe-2fdc-4225-8f15-2d294389e152-kube-api-access-26mr5\") pod \"344851fe-2fdc-4225-8f15-2d294389e152\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") "
Sep 06 05:27:08.560405 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.560408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-metrics-certs\") pod \"344851fe-2fdc-4225-8f15-2d294389e152\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") "
Sep 06 05:27:08.560581 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.560445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344851fe-2fdc-4225-8f15-2d294389e152-openshift-service-ca\") pod \"344851fe-2fdc-4225-8f15-2d294389e152\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") "
Sep 06 05:27:08.560581 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.560479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-maas-api-tls\") pod \"344851fe-2fdc-4225-8f15-2d294389e152\" (UID: \"344851fe-2fdc-4225-8f15-2d294389e152\") "
Sep 06 05:27:08.560862 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.560838    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344851fe-2fdc-4225-8f15-2d294389e152-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "344851fe-2fdc-4225-8f15-2d294389e152" (UID: "344851fe-2fdc-4225-8f15-2d294389e152"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:27:08.562666 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.562641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "344851fe-2fdc-4225-8f15-2d294389e152" (UID: "344851fe-2fdc-4225-8f15-2d294389e152"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:27:08.562752 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.562676    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "344851fe-2fdc-4225-8f15-2d294389e152" (UID: "344851fe-2fdc-4225-8f15-2d294389e152"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:27:08.562802 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.562786    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344851fe-2fdc-4225-8f15-2d294389e152-kube-api-access-26mr5" (OuterVolumeSpecName: "kube-api-access-26mr5") pod "344851fe-2fdc-4225-8f15-2d294389e152" (UID: "344851fe-2fdc-4225-8f15-2d294389e152"). InnerVolumeSpecName "kube-api-access-26mr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:27:08.661462 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.661438    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:08.661462 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.661459    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26mr5\" (UniqueName: \"kubernetes.io/projected/344851fe-2fdc-4225-8f15-2d294389e152-kube-api-access-26mr5\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:08.661665 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.661471    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344851fe-2fdc-4225-8f15-2d294389e152-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:08.661665 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:08.661480    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344851fe-2fdc-4225-8f15-2d294389e152-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:09.251007 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:09.250974    2580 generic.go:356] "Generic (PLEG): container finished" podID="344851fe-2fdc-4225-8f15-2d294389e152" containerID="df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd" exitCode=0
Sep 06 05:27:09.251478 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:09.251021    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"
Sep 06 05:27:09.251478 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:09.251056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg" event={"ID":"344851fe-2fdc-4225-8f15-2d294389e152","Type":"ContainerDied","Data":"df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd"}
Sep 06 05:27:09.251478 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:09.251109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg" event={"ID":"344851fe-2fdc-4225-8f15-2d294389e152","Type":"ContainerDied","Data":"f56d59c20c8bee88f1939452a4dbd24ae736c06a588e49b8e628f2026b5c5277"}
Sep 06 05:27:09.251478 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:09.251131    2580 scope.go:117] "RemoveContainer" containerID="df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd"
Sep 06 05:27:09.261719 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:09.261702    2580 scope.go:117] "RemoveContainer" containerID="df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd"
Sep 06 05:27:09.261976 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:27:09.261957    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd\": container with ID starting with df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd not found: ID does not exist" containerID="df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd"
Sep 06 05:27:09.262036 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:09.261983    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd"} err="failed to get container status \"df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd\": rpc error: code = NotFound desc = could not find container \"df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd\": container with ID starting with df1396f858fb9b884a83e44c84aa4695a5f8c1c1a35481b3b6d57eeb964f1ebd not found: ID does not exist"
Sep 06 05:27:09.271918 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:09.271893    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"]
Sep 06 05:27:09.276467 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:09.276448    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-22321a-74b96c55bb-nktpg"]
Sep 06 05:27:10.352347 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:10.349147    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="344851fe-2fdc-4225-8f15-2d294389e152" path="/var/lib/kubelet/pods/344851fe-2fdc-4225-8f15-2d294389e152/volumes"
Sep 06 05:27:11.543413 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:11.543372    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:27:25.061429 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.061396    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"]
Sep 06 05:27:25.061880 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.061598    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk" podUID="5c42e1b0-411e-42f9-99fb-284999098266" containerName="maas-api" containerID="cri-o://43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab" gracePeriod=30
Sep 06 05:27:25.307012 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.306984    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:27:25.343020 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.342941    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c42e1b0-411e-42f9-99fb-284999098266" containerID="43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab" exitCode=0
Sep 06 05:27:25.343020 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.342986    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"
Sep 06 05:27:25.343237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.343031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk" event={"ID":"5c42e1b0-411e-42f9-99fb-284999098266","Type":"ContainerDied","Data":"43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab"}
Sep 06 05:27:25.343237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.343077    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk" event={"ID":"5c42e1b0-411e-42f9-99fb-284999098266","Type":"ContainerDied","Data":"11a1f695ddba4a9c6121700638704da568dee73511b320420783cad4464a16ec"}
Sep 06 05:27:25.343237 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.343106    2580 scope.go:117] "RemoveContainer" containerID="43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab"
Sep 06 05:27:25.352055 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.352040    2580 scope.go:117] "RemoveContainer" containerID="43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab"
Sep 06 05:27:25.352354 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:27:25.352329    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab\": container with ID starting with 43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab not found: ID does not exist" containerID="43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab"
Sep 06 05:27:25.352409 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.352364    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab"} err="failed to get container status \"43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab\": rpc error: code = NotFound desc = could not find container \"43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab\": container with ID starting with 43a2cae07e48b32240b950191413f349b56db3a038bc09ae6021487192391dab not found: ID does not exist"
Sep 06 05:27:25.503832 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.503800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-metrics-certs\") pod \"5c42e1b0-411e-42f9-99fb-284999098266\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") "
Sep 06 05:27:25.503998 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.503885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-maas-api-tls\") pod \"5c42e1b0-411e-42f9-99fb-284999098266\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") "
Sep 06 05:27:25.503998 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.503930    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c42e1b0-411e-42f9-99fb-284999098266-openshift-service-ca\") pod \"5c42e1b0-411e-42f9-99fb-284999098266\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") "
Sep 06 05:27:25.503998 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.503965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfxlb\" (UniqueName: \"kubernetes.io/projected/5c42e1b0-411e-42f9-99fb-284999098266-kube-api-access-lfxlb\") pod \"5c42e1b0-411e-42f9-99fb-284999098266\" (UID: \"5c42e1b0-411e-42f9-99fb-284999098266\") "
Sep 06 05:27:25.504369 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.504331    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c42e1b0-411e-42f9-99fb-284999098266-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c42e1b0-411e-42f9-99fb-284999098266" (UID: "5c42e1b0-411e-42f9-99fb-284999098266"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:27:25.506139 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.506106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c42e1b0-411e-42f9-99fb-284999098266" (UID: "5c42e1b0-411e-42f9-99fb-284999098266"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:27:25.506139 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.506123    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5c42e1b0-411e-42f9-99fb-284999098266" (UID: "5c42e1b0-411e-42f9-99fb-284999098266"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:27:25.506305 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.506187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c42e1b0-411e-42f9-99fb-284999098266-kube-api-access-lfxlb" (OuterVolumeSpecName: "kube-api-access-lfxlb") pod "5c42e1b0-411e-42f9-99fb-284999098266" (UID: "5c42e1b0-411e-42f9-99fb-284999098266"). InnerVolumeSpecName "kube-api-access-lfxlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:27:25.604822 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.604753    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c42e1b0-411e-42f9-99fb-284999098266-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:25.604822 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.604777    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfxlb\" (UniqueName: \"kubernetes.io/projected/5c42e1b0-411e-42f9-99fb-284999098266-kube-api-access-lfxlb\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:25.604822 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.604786    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:25.604822 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.604794    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c42e1b0-411e-42f9-99fb-284999098266-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:25.664343 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.664312    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"]
Sep 06 05:27:25.668047 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:25.668020    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-494740-659c5b49c7-ztkzk"]
Sep 06 05:27:26.347780 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:26.347748    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c42e1b0-411e-42f9-99fb-284999098266" path="/var/lib/kubelet/pods/5c42e1b0-411e-42f9-99fb-284999098266/volumes"
Sep 06 05:27:28.045732 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:28.045697    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:27:42.575454 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.575422    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"]
Sep 06 05:27:42.575856 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.575644    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb" podUID="aae78a95-1862-4947-8a32-6a1ba76ebf98" containerName="maas-api" containerID="cri-o://e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0" gracePeriod=30
Sep 06 05:27:42.823424 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.823399    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:27:42.839789 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.839720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae78a95-1862-4947-8a32-6a1ba76ebf98-openshift-service-ca\") pod \"aae78a95-1862-4947-8a32-6a1ba76ebf98\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") "
Sep 06 05:27:42.839789 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.839764    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-metrics-certs\") pod \"aae78a95-1862-4947-8a32-6a1ba76ebf98\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") "
Sep 06 05:27:42.839974 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.839791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4khcr\" (UniqueName: \"kubernetes.io/projected/aae78a95-1862-4947-8a32-6a1ba76ebf98-kube-api-access-4khcr\") pod \"aae78a95-1862-4947-8a32-6a1ba76ebf98\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") "
Sep 06 05:27:42.839974 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.839826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-maas-api-tls\") pod \"aae78a95-1862-4947-8a32-6a1ba76ebf98\" (UID: \"aae78a95-1862-4947-8a32-6a1ba76ebf98\") "
Sep 06 05:27:42.840089 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.840068    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae78a95-1862-4947-8a32-6a1ba76ebf98-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aae78a95-1862-4947-8a32-6a1ba76ebf98" (UID: "aae78a95-1862-4947-8a32-6a1ba76ebf98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:27:42.841605 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.840221    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae78a95-1862-4947-8a32-6a1ba76ebf98-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:42.843033 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.842897    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aae78a95-1862-4947-8a32-6a1ba76ebf98" (UID: "aae78a95-1862-4947-8a32-6a1ba76ebf98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:27:42.843033 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.842956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae78a95-1862-4947-8a32-6a1ba76ebf98-kube-api-access-4khcr" (OuterVolumeSpecName: "kube-api-access-4khcr") pod "aae78a95-1862-4947-8a32-6a1ba76ebf98" (UID: "aae78a95-1862-4947-8a32-6a1ba76ebf98"). InnerVolumeSpecName "kube-api-access-4khcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:27:42.843033 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.842940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aae78a95-1862-4947-8a32-6a1ba76ebf98" (UID: "aae78a95-1862-4947-8a32-6a1ba76ebf98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:27:42.941248 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.941221    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:42.941248 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.941244    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4khcr\" (UniqueName: \"kubernetes.io/projected/aae78a95-1862-4947-8a32-6a1ba76ebf98-kube-api-access-4khcr\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:42.941389 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:42.941254    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aae78a95-1862-4947-8a32-6a1ba76ebf98-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:27:43.451042 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:43.451006    2580 generic.go:356] "Generic (PLEG): container finished" podID="aae78a95-1862-4947-8a32-6a1ba76ebf98" containerID="e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0" exitCode=0
Sep 06 05:27:43.451287 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:43.451058    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"
Sep 06 05:27:43.451287 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:43.451103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb" event={"ID":"aae78a95-1862-4947-8a32-6a1ba76ebf98","Type":"ContainerDied","Data":"e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0"}
Sep 06 05:27:43.451287 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:43.451217    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb" event={"ID":"aae78a95-1862-4947-8a32-6a1ba76ebf98","Type":"ContainerDied","Data":"b110d5e41e05f9281f2bf002c6b862ba8e44cdc66acbfa2487a59307688e508f"}
Sep 06 05:27:43.451287 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:43.451251    2580 scope.go:117] "RemoveContainer" containerID="e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0"
Sep 06 05:27:43.460987 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:43.460972    2580 scope.go:117] "RemoveContainer" containerID="e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0"
Sep 06 05:27:43.461249 ip-10-0-128-168 kubenswrapper[2580]: E0906 05:27:43.461225    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0\": container with ID starting with e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0 not found: ID does not exist" containerID="e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0"
Sep 06 05:27:43.461304 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:43.461260    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0"} err="failed to get container status \"e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0\": rpc error: code = NotFound desc = could not find container \"e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0\": container with ID starting with e1f4c201111a9eaa85ba19cebedf521a19b64206180b5ab0e578b5664541bcf0 not found: ID does not exist"
Sep 06 05:27:43.471852 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:43.471828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"]
Sep 06 05:27:43.475572 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:43.475553    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-167b5b-846b56896c-jl7kb"]
Sep 06 05:27:44.347701 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:44.347668    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aae78a95-1862-4947-8a32-6a1ba76ebf98" path="/var/lib/kubelet/pods/aae78a95-1862-4947-8a32-6a1ba76ebf98/volumes"
Sep 06 05:27:45.137446 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:45.137414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:27:53.443458 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:27:53.443426    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:28:24.701972 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:28:24.701901    2580 scope.go:117] "RemoveContainer" containerID="25e53072c921b9323d8b69bf360482d8d656e282a655a76455c6b5d02eefb0ac"
Sep 06 05:29:14.644982 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:29:14.644947    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:29:40.448583 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:29:40.448548    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:29:47.438221 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:29:47.438191    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:29:57.742800 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:29:57.742764    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:30:15.340109 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:30:15.340076    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:30:22.042508 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:30:22.042468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:30:24.561556 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:30:24.561526    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:30:24.566599 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:30:24.566580    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:31:49.646225 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:31:49.646177    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:31:56.538211 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:31:56.538174    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:32:02.638734 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:32:02.638694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:32:24.243668 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:32:24.243636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:32:33.003622 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:32:33.003589    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:32:42.547760 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:32:42.547730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:32:51.640502 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:32:51.640467    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:32:59.845863 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:32:59.845829    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:33:00.273888 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.273856    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"]
Sep 06 05:33:00.274325 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274308    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aae78a95-1862-4947-8a32-6a1ba76ebf98" containerName="maas-api"
Sep 06 05:33:00.274440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274327    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aae78a95-1862-4947-8a32-6a1ba76ebf98" containerName="maas-api"
Sep 06 05:33:00.274440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274340    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="344851fe-2fdc-4225-8f15-2d294389e152" containerName="maas-api"
Sep 06 05:33:00.274440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274347    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="344851fe-2fdc-4225-8f15-2d294389e152" containerName="maas-api"
Sep 06 05:33:00.274440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274377    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e2be20-6340-482f-89e4-5379bb84b016" containerName="main"
Sep 06 05:33:00.274440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274386    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e2be20-6340-482f-89e4-5379bb84b016" containerName="main"
Sep 06 05:33:00.274440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274403    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c42e1b0-411e-42f9-99fb-284999098266" containerName="maas-api"
Sep 06 05:33:00.274440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274411    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c42e1b0-411e-42f9-99fb-284999098266" containerName="maas-api"
Sep 06 05:33:00.274440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274423    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19c6cde-4aa7-4c42-8d0c-cd7106b66b71" containerName="maas-api"
Sep 06 05:33:00.274440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274431    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19c6cde-4aa7-4c42-8d0c-cd7106b66b71" containerName="maas-api"
Sep 06 05:33:00.274900 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274526    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="344851fe-2fdc-4225-8f15-2d294389e152" containerName="maas-api"
Sep 06 05:33:00.274900 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274537    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c42e1b0-411e-42f9-99fb-284999098266" containerName="maas-api"
Sep 06 05:33:00.274900 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274545    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e2be20-6340-482f-89e4-5379bb84b016" containerName="main"
Sep 06 05:33:00.274900 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274554    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d19c6cde-4aa7-4c42-8d0c-cd7106b66b71" containerName="maas-api"
Sep 06 05:33:00.274900 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.274566    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aae78a95-1862-4947-8a32-6a1ba76ebf98" containerName="maas-api"
Sep 06 05:33:00.277759 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.277739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.281532 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.281508    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 05:33:00.281663 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.281626    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 05:33:00.281663 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.281649    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 06 05:33:00.281813 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.281683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d2kzq\""
Sep 06 05:33:00.281878 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.281849    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 05:33:00.285911 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.285616    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"]
Sep 06 05:33:00.437102 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.437076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-metrics-certs\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.437280 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.437117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-maas-api-tls\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.437280 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.437219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6fws\" (UniqueName: \"kubernetes.io/projected/30404b81-52e8-48be-88d1-c0005c64ee17-kube-api-access-m6fws\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.437280 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.437244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30404b81-52e8-48be-88d1-c0005c64ee17-openshift-service-ca\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.537839 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.537755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6fws\" (UniqueName: \"kubernetes.io/projected/30404b81-52e8-48be-88d1-c0005c64ee17-kube-api-access-m6fws\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.537839 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.537789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30404b81-52e8-48be-88d1-c0005c64ee17-openshift-service-ca\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.538028 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.537856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-metrics-certs\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.538028 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.537903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-maas-api-tls\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.538507 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.538480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30404b81-52e8-48be-88d1-c0005c64ee17-openshift-service-ca\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.540547 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.540526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-maas-api-tls\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.540660 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.540560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-metrics-certs\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.544837 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.544812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6fws\" (UniqueName: \"kubernetes.io/projected/30404b81-52e8-48be-88d1-c0005c64ee17-kube-api-access-m6fws\") pod \"maas-api-7b97c6dbfb-tmbvd\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.590593 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.590559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:00.862870 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.862840    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"]
Sep 06 05:33:00.864480 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:33:00.864456    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30404b81_52e8_48be_88d1_c0005c64ee17.slice/crio-3aee5e26b7e37201b4b522f0400c41ce8c7613cbb55ef12ada961198893b26d4 WatchSource:0}: Error finding container 3aee5e26b7e37201b4b522f0400c41ce8c7613cbb55ef12ada961198893b26d4: Status 404 returned error can't find the container with id 3aee5e26b7e37201b4b522f0400c41ce8c7613cbb55ef12ada961198893b26d4
Sep 06 05:33:00.866170 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:00.866141    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 05:33:01.153182 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:01.153088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd" event={"ID":"30404b81-52e8-48be-88d1-c0005c64ee17","Type":"ContainerStarted","Data":"3aee5e26b7e37201b4b522f0400c41ce8c7613cbb55ef12ada961198893b26d4"}
Sep 06 05:33:02.159514 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:02.159481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd" event={"ID":"30404b81-52e8-48be-88d1-c0005c64ee17","Type":"ContainerStarted","Data":"d972c5c25ec6a8d1941bd1fc96310ed77685cc96179356a7a2e3d01788c2a69e"}
Sep 06 05:33:02.159904 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:02.159587    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:02.179596 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:02.179551    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd" podStartSLOduration=1.7110776090000002 podStartE2EDuration="2.179537107s" totalImagesPullingTime="468.459498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:33:02.176502311 +0000 UTC m=+2258.318062263" watchObservedRunningTime="2026-09-06 05:33:02.179537107 +0000 UTC m=+2258.321097075"
Sep 06 05:33:07.738440 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:07.738406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:33:07.964647 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:07.964615    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"]
Sep 06 05:33:07.964858 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:07.964834    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd" podUID="30404b81-52e8-48be-88d1-c0005c64ee17" containerName="maas-api" containerID="cri-o://d972c5c25ec6a8d1941bd1fc96310ed77685cc96179356a7a2e3d01788c2a69e" gracePeriod=30
Sep 06 05:33:07.969725 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:07.969704    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:07.993695 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:07.993640    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"]
Sep 06 05:33:07.997137 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:07.997114    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.005359 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.005335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"]
Sep 06 05:33:08.096050 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.096025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41708f5c-f30d-4c91-a824-158c3f9806d2-openshift-service-ca\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.096201 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.096058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41708f5c-f30d-4c91-a824-158c3f9806d2-metrics-certs\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.096201 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.096097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41708f5c-f30d-4c91-a824-158c3f9806d2-maas-api-tls\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.096201 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.096197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqv76\" (UniqueName: \"kubernetes.io/projected/41708f5c-f30d-4c91-a824-158c3f9806d2-kube-api-access-cqv76\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.193443 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.193415    2580 generic.go:356] "Generic (PLEG): container finished" podID="30404b81-52e8-48be-88d1-c0005c64ee17" containerID="d972c5c25ec6a8d1941bd1fc96310ed77685cc96179356a7a2e3d01788c2a69e" exitCode=0
Sep 06 05:33:08.193579 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.193483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd" event={"ID":"30404b81-52e8-48be-88d1-c0005c64ee17","Type":"ContainerDied","Data":"d972c5c25ec6a8d1941bd1fc96310ed77685cc96179356a7a2e3d01788c2a69e"}
Sep 06 05:33:08.196776 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.196758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41708f5c-f30d-4c91-a824-158c3f9806d2-openshift-service-ca\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.196840 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.196787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41708f5c-f30d-4c91-a824-158c3f9806d2-metrics-certs\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.196840 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.196818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41708f5c-f30d-4c91-a824-158c3f9806d2-maas-api-tls\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.196941 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.196888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqv76\" (UniqueName: \"kubernetes.io/projected/41708f5c-f30d-4c91-a824-158c3f9806d2-kube-api-access-cqv76\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.197501 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.197475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41708f5c-f30d-4c91-a824-158c3f9806d2-openshift-service-ca\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.200008 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.199991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41708f5c-f30d-4c91-a824-158c3f9806d2-maas-api-tls\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.200168 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.200132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41708f5c-f30d-4c91-a824-158c3f9806d2-metrics-certs\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.204337 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.204318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqv76\" (UniqueName: \"kubernetes.io/projected/41708f5c-f30d-4c91-a824-158c3f9806d2-kube-api-access-cqv76\") pod \"maas-api-6886844f88-5fwcq\" (UID: \"41708f5c-f30d-4c91-a824-158c3f9806d2\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.222032 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.222015    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:08.297693 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.297633    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-maas-api-tls\") pod \"30404b81-52e8-48be-88d1-c0005c64ee17\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") "
Sep 06 05:33:08.297791 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.297690    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30404b81-52e8-48be-88d1-c0005c64ee17-openshift-service-ca\") pod \"30404b81-52e8-48be-88d1-c0005c64ee17\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") "
Sep 06 05:33:08.297791 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.297735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6fws\" (UniqueName: \"kubernetes.io/projected/30404b81-52e8-48be-88d1-c0005c64ee17-kube-api-access-m6fws\") pod \"30404b81-52e8-48be-88d1-c0005c64ee17\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") "
Sep 06 05:33:08.297907 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.297791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-metrics-certs\") pod \"30404b81-52e8-48be-88d1-c0005c64ee17\" (UID: \"30404b81-52e8-48be-88d1-c0005c64ee17\") "
Sep 06 05:33:08.298085 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.298055    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30404b81-52e8-48be-88d1-c0005c64ee17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30404b81-52e8-48be-88d1-c0005c64ee17" (UID: "30404b81-52e8-48be-88d1-c0005c64ee17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 05:33:08.299882 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.299861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30404b81-52e8-48be-88d1-c0005c64ee17" (UID: "30404b81-52e8-48be-88d1-c0005c64ee17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:33:08.299976 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.299916    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "30404b81-52e8-48be-88d1-c0005c64ee17" (UID: "30404b81-52e8-48be-88d1-c0005c64ee17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 05:33:08.299976 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.299934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30404b81-52e8-48be-88d1-c0005c64ee17-kube-api-access-m6fws" (OuterVolumeSpecName: "kube-api-access-m6fws") pod "30404b81-52e8-48be-88d1-c0005c64ee17" (UID: "30404b81-52e8-48be-88d1-c0005c64ee17"). InnerVolumeSpecName "kube-api-access-m6fws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 05:33:08.350717 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.350700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:08.398881 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.398851    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6fws\" (UniqueName: \"kubernetes.io/projected/30404b81-52e8-48be-88d1-c0005c64ee17-kube-api-access-m6fws\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:33:08.399012 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.398886    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-metrics-certs\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:33:08.399012 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.398904    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30404b81-52e8-48be-88d1-c0005c64ee17-maas-api-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:33:08.399012 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.398916    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30404b81-52e8-48be-88d1-c0005c64ee17-openshift-service-ca\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Sep 06 05:33:08.621471 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:08.621442    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"]
Sep 06 05:33:08.622815 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:33:08.622790    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41708f5c_f30d_4c91_a824_158c3f9806d2.slice/crio-786ab2c46be2b8985619fb0da9332abd225d39e2e9ce8632783b157ac3235c00 WatchSource:0}: Error finding container 786ab2c46be2b8985619fb0da9332abd225d39e2e9ce8632783b157ac3235c00: Status 404 returned error can't find the container with id 786ab2c46be2b8985619fb0da9332abd225d39e2e9ce8632783b157ac3235c00
Sep 06 05:33:09.201784 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:09.201294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq" event={"ID":"41708f5c-f30d-4c91-a824-158c3f9806d2","Type":"ContainerStarted","Data":"643d278c498c272abcafc1df8711cb567fad71268ca52642b98d30a8dcad11f1"}
Sep 06 05:33:09.202255 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:09.201779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq" event={"ID":"41708f5c-f30d-4c91-a824-158c3f9806d2","Type":"ContainerStarted","Data":"786ab2c46be2b8985619fb0da9332abd225d39e2e9ce8632783b157ac3235c00"}
Sep 06 05:33:09.202255 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:09.201831    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:09.205790 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:09.205762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd" event={"ID":"30404b81-52e8-48be-88d1-c0005c64ee17","Type":"ContainerDied","Data":"3aee5e26b7e37201b4b522f0400c41ce8c7613cbb55ef12ada961198893b26d4"}
Sep 06 05:33:09.205932 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:09.205852    2580 scope.go:117] "RemoveContainer" containerID="d972c5c25ec6a8d1941bd1fc96310ed77685cc96179356a7a2e3d01788c2a69e"
Sep 06 05:33:09.205932 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:09.205887    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"
Sep 06 05:33:09.216926 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:09.216889    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq" podStartSLOduration=1.9075481779999999 podStartE2EDuration="2.216876579s" totalImagesPullingTime="309.328401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:33:09.215731823 +0000 UTC m=+2265.357291801" watchObservedRunningTime="2026-09-06 05:33:09.216876579 +0000 UTC m=+2265.358436530"
Sep 06 05:33:09.229082 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:09.229059    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"]
Sep 06 05:33:09.232168 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:09.232133    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-tmbvd"]
Sep 06 05:33:10.348407 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:10.348375    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30404b81-52e8-48be-88d1-c0005c64ee17" path="/var/lib/kubelet/pods/30404b81-52e8-48be-88d1-c0005c64ee17/volumes"
Sep 06 05:33:15.218289 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:15.218254    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6886844f88-5fwcq"
Sep 06 05:33:15.543138 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:15.543057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:33:23.745590 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:23.745561    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:33:31.547633 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:31.543006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-vr8tg"]
Sep 06 05:33:33.355359 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:33.355325    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7bcd7bc469-r2bf5_9195228f-e128-43fa-883d-e1a20a859f34/authorino/0.log"
Sep 06 05:33:39.511927 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:39.511901    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657c48fd97-ndrr9_50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35/copy-manifests/0.log"
Sep 06 05:33:39.518977 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:39.518953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657c48fd97-ndrr9_50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35/manager/0.log"
Sep 06 05:33:40.000385 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:40.000357    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6886844f88-5fwcq_41708f5c-f30d-4c91-a824-158c3f9806d2/maas-api/0.log"
Sep 06 05:33:41.194424 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:41.194390    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7bcd7bc469-r2bf5_9195228f-e128-43fa-883d-e1a20a859f34/authorino/0.log"
Sep 06 05:33:41.521305 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:41.521281    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-rjg5t_7c3b7076-9e38-4518-88aa-4e8f62b1554a/kuadrant-console-plugin/0.log"
Sep 06 05:33:41.762235 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:41.762208    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-vr8tg_29570032-d53c-48cc-95b0-857a2933b80b/limitador/0.log"
Sep 06 05:33:42.441384 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:42.441353    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c96787df8-bmmvf_1fa736a2-0e9f-4018-add2-449fdf2d48c8/kube-auth-proxy/0.log"
Sep 06 05:33:43.599078 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:43.599053    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-nwshq_2fb801d7-cd7e-4371-b19f-57625e399e3e/main/0.log"
Sep 06 05:33:43.709492 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:43.709465    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-8vkxs_335f0e6d-7d90-4912-8b01-0a3b6c76fe88/main/0.log"
Sep 06 05:33:51.902849 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:51.902817    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g6f2s_d08567e7-7463-408b-9d6f-d6edca57b781/global-pull-secret-syncer/0.log"
Sep 06 05:33:52.071754 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:52.071720    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d24f8_2153f522-a60a-4c12-8aea-bf25f12e162e/konnectivity-agent/0.log"
Sep 06 05:33:52.132076 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:52.132048    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-168.ec2.internal_2a6422a356691d2cbb645b3d74be5d53/haproxy/0.log"
Sep 06 05:33:56.901604 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:56.901573    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7bcd7bc469-r2bf5_9195228f-e128-43fa-883d-e1a20a859f34/authorino/0.log"
Sep 06 05:33:56.980521 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:56.980486    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-rjg5t_7c3b7076-9e38-4518-88aa-4e8f62b1554a/kuadrant-console-plugin/0.log"
Sep 06 05:33:57.038852 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:57.038823    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-vr8tg_29570032-d53c-48cc-95b0-857a2933b80b/limitador/0.log"
Sep 06 05:33:58.704677 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:58.704648    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gsqlq_bf388998-929b-40b8-99a0-0a9ed282507a/kube-state-metrics/0.log"
Sep 06 05:33:58.718172 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:58.718125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gsqlq_bf388998-929b-40b8-99a0-0a9ed282507a/kube-rbac-proxy-main/0.log"
Sep 06 05:33:58.735755 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:58.735728    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gsqlq_bf388998-929b-40b8-99a0-0a9ed282507a/kube-rbac-proxy-self/0.log"
Sep 06 05:33:58.765989 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:58.765967    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b64f578b7-8zvff_c25ce7ff-6a88-4eba-8c76-81096a80ca14/metrics-server/0.log"
Sep 06 05:33:58.801021 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:58.800996    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-wnlzt_8bbef036-9031-4ad0-a729-438c9ff26410/monitoring-plugin/0.log"
Sep 06 05:33:58.826940 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:58.826903    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zj4c_9fe134b3-19d7-4321-99ba-1addfd2cfae0/node-exporter/0.log"
Sep 06 05:33:58.849967 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:58.849938    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zj4c_9fe134b3-19d7-4321-99ba-1addfd2cfae0/kube-rbac-proxy/0.log"
Sep 06 05:33:58.868117 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:58.868092    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zj4c_9fe134b3-19d7-4321-99ba-1addfd2cfae0/init-textfile/0.log"
Sep 06 05:33:59.022938 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:59.022911    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xgzf2_e91beee9-c6b0-4164-9751-5b85b4c08ab0/kube-rbac-proxy-main/0.log"
Sep 06 05:33:59.045941 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:59.045915    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xgzf2_e91beee9-c6b0-4164-9751-5b85b4c08ab0/kube-rbac-proxy-self/0.log"
Sep 06 05:33:59.077069 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:59.077025    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xgzf2_e91beee9-c6b0-4164-9751-5b85b4c08ab0/openshift-state-metrics/0.log"
Sep 06 05:33:59.314555 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:59.314443    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-rw6zb_6dc1e05a-c5d6-4b32-8a84-bab53f3c439f/prometheus-operator/0.log"
Sep 06 05:33:59.329782 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:59.329743    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-rw6zb_6dc1e05a-c5d6-4b32-8a84-bab53f3c439f/kube-rbac-proxy/0.log"
Sep 06 05:33:59.391240 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:59.391217    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ffd9c6b55-4s8q7_54a756ad-0150-4934-a646-14f5ed91d82e/telemeter-client/0.log"
Sep 06 05:33:59.408789 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:59.408763    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ffd9c6b55-4s8q7_54a756ad-0150-4934-a646-14f5ed91d82e/reload/0.log"
Sep 06 05:33:59.428806 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:33:59.428785    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ffd9c6b55-4s8q7_54a756ad-0150-4934-a646-14f5ed91d82e/kube-rbac-proxy/0.log"
Sep 06 05:34:00.980039 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:00.980006    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-dtkkv_c64b1fde-c4e4-49b4-a005-7e721932b165/networking-console-plugin/0.log"
Sep 06 05:34:02.145097 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.145056    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"]
Sep 06 05:34:02.145667 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.145645    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30404b81-52e8-48be-88d1-c0005c64ee17" containerName="maas-api"
Sep 06 05:34:02.145763 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.145670    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="30404b81-52e8-48be-88d1-c0005c64ee17" containerName="maas-api"
Sep 06 05:34:02.145817 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.145778    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="30404b81-52e8-48be-88d1-c0005c64ee17" containerName="maas-api"
Sep 06 05:34:02.149253 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.149040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.151226 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.151203    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5ns8q\"/\"openshift-service-ca.crt\""
Sep 06 05:34:02.151369 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.151252    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5ns8q\"/\"default-dockercfg-p7szt\""
Sep 06 05:34:02.152079 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.152060    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5ns8q\"/\"kube-root-ca.crt\""
Sep 06 05:34:02.158135 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.158111    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"]
Sep 06 05:34:02.251197 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.251143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-proc\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.251340 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.251284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp8rm\" (UniqueName: \"kubernetes.io/projected/6c162eb6-e5ca-415c-852f-60c27780564b-kube-api-access-mp8rm\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.251422 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.251343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-sys\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.251422 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.251382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-podres\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.251524 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.251416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-lib-modules\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.352492 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.352468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-proc\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.352600 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.352529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp8rm\" (UniqueName: \"kubernetes.io/projected/6c162eb6-e5ca-415c-852f-60c27780564b-kube-api-access-mp8rm\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.352600 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.352552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-sys\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.352600 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.352571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-podres\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.352600 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.352585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-proc\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.352803 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.352588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-lib-modules\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.352803 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.352653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-sys\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.352803 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.352674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-podres\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.352803 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.352731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c162eb6-e5ca-415c-852f-60c27780564b-lib-modules\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.359830 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.359812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp8rm\" (UniqueName: \"kubernetes.io/projected/6c162eb6-e5ca-415c-852f-60c27780564b-kube-api-access-mp8rm\") pod \"perf-node-gather-daemonset-hvfvk\" (UID: \"6c162eb6-e5ca-415c-852f-60c27780564b\") " pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.460003 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.459943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:02.729562 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:02.729534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"]
Sep 06 05:34:02.731410 ip-10-0-128-168 kubenswrapper[2580]: W0906 05:34:02.731385    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6c162eb6_e5ca_415c_852f_60c27780564b.slice/crio-5851cc8289d367f026def8dcc68a309fc078c4c004593b59945b85c1b1750b13 WatchSource:0}: Error finding container 5851cc8289d367f026def8dcc68a309fc078c4c004593b59945b85c1b1750b13: Status 404 returned error can't find the container with id 5851cc8289d367f026def8dcc68a309fc078c4c004593b59945b85c1b1750b13
Sep 06 05:34:03.499886 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:03.499832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk" event={"ID":"6c162eb6-e5ca-415c-852f-60c27780564b","Type":"ContainerStarted","Data":"00bae5134e50e16138c6d0fe1a9c7996b345d3ca24609ce90d1fb89a85ac6519"}
Sep 06 05:34:03.500306 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:03.499907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk" event={"ID":"6c162eb6-e5ca-415c-852f-60c27780564b","Type":"ContainerStarted","Data":"5851cc8289d367f026def8dcc68a309fc078c4c004593b59945b85c1b1750b13"}
Sep 06 05:34:03.500306 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:03.500069    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:03.515314 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:03.515270    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk" podStartSLOduration=1.515255977 podStartE2EDuration="1.515255977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:34:03.512576658 +0000 UTC m=+2319.654136612" watchObservedRunningTime="2026-09-06 05:34:03.515255977 +0000 UTC m=+2319.656815919"
Sep 06 05:34:03.990093 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:03.990070    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mx4tj_84a76bc6-6f0f-4d5c-b48f-9e8ab6651556/dns/0.log"
Sep 06 05:34:04.007879 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:04.007859    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mx4tj_84a76bc6-6f0f-4d5c-b48f-9e8ab6651556/kube-rbac-proxy/0.log"
Sep 06 05:34:04.109350 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:04.109325    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kwqp6_fcfa2398-7840-4bc9-b5d1-61bc9e8cfc96/dns-node-resolver/0.log"
Sep 06 05:34:04.646104 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:04.646069    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hr7nb_53aa3ac4-3e77-40df-9136-ff0e41c2606b/node-ca/0.log"
Sep 06 05:34:05.818229 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:05.818186    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c96787df8-bmmvf_1fa736a2-0e9f-4018-add2-449fdf2d48c8/kube-auth-proxy/0.log"
Sep 06 05:34:06.783495 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:06.783468    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-55l5h_8b09e0a1-fcf5-475c-bbfa-66fcf9c7eff3/serve-healthcheck-canary/0.log"
Sep 06 05:34:07.429623 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:07.429585    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rq4r9_f211ba82-097b-48c4-8449-a43018d3bcda/kube-rbac-proxy/0.log"
Sep 06 05:34:07.462391 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:07.462367    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rq4r9_f211ba82-097b-48c4-8449-a43018d3bcda/exporter/0.log"
Sep 06 05:34:07.482441 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:07.482409    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rq4r9_f211ba82-097b-48c4-8449-a43018d3bcda/extractor/0.log"
Sep 06 05:34:09.516008 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:09.515974    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5ns8q/perf-node-gather-daemonset-hvfvk"
Sep 06 05:34:10.196059 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:10.196023    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657c48fd97-ndrr9_50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35/manager/0.log"
Sep 06 05:34:10.213349 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:10.213326    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657c48fd97-ndrr9_50b1b2e0-9d6e-4a4e-a0ef-5128b8a95d35/copy-manifests/0.log"
Sep 06 05:34:11.590904 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:11.590876    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7d8d45985c-gsjmd_3f3c2667-c372-4159-97e5-369c98275a03/manager/0.log"
Sep 06 05:34:13.142794 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:13.142763    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c96787df8-bmmvf_1fa736a2-0e9f-4018-add2-449fdf2d48c8/kube-auth-proxy/0.log"
Sep 06 05:34:15.167972 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:15.167938    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c96787df8-bmmvf_1fa736a2-0e9f-4018-add2-449fdf2d48c8/kube-auth-proxy/0.log"
Sep 06 05:34:16.780450 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:16.780314    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5kpkm_210495ce-119c-4189-8b5d-72a22d362b66/migrator/0.log"
Sep 06 05:34:16.799993 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:16.799966    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5kpkm_210495ce-119c-4189-8b5d-72a22d362b66/graceful-termination/0.log"
Sep 06 05:34:17.154610 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:17.154524    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-x6wn5_a21526c5-2c0d-48dc-89c1-11b2abd6dac3/kube-storage-version-migrator-operator/1.log"
Sep 06 05:34:17.156176 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:17.156136    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-x6wn5_a21526c5-2c0d-48dc-89c1-11b2abd6dac3/kube-storage-version-migrator-operator/0.log"
Sep 06 05:34:18.108670 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:18.108641    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4568p_a99b800d-76a9-4349-82f8-55804daefcde/kube-multus-additional-cni-plugins/0.log"
Sep 06 05:34:18.126558 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:18.126532    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4568p_a99b800d-76a9-4349-82f8-55804daefcde/egress-router-binary-copy/0.log"
Sep 06 05:34:18.142872 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:18.142802    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4568p_a99b800d-76a9-4349-82f8-55804daefcde/cni-plugins/0.log"
Sep 06 05:34:18.160921 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:18.160890    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4568p_a99b800d-76a9-4349-82f8-55804daefcde/bond-cni-plugin/0.log"
Sep 06 05:34:18.178287 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:18.178267    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4568p_a99b800d-76a9-4349-82f8-55804daefcde/routeoverride-cni/0.log"
Sep 06 05:34:18.194965 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:18.194945    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4568p_a99b800d-76a9-4349-82f8-55804daefcde/whereabouts-cni-bincopy/0.log"
Sep 06 05:34:18.212230 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:18.212208    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4568p_a99b800d-76a9-4349-82f8-55804daefcde/whereabouts-cni/0.log"
Sep 06 05:34:18.583037 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:18.583009    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vrnfr_ebf6fbec-f0bf-44ba-99dd-355169f9da5e/kube-multus/0.log"
Sep 06 05:34:18.603764 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:18.603742    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-csrcb_d3a5a3f7-dc34-4f63-9fa9-5660515f8000/network-metrics-daemon/0.log"
Sep 06 05:34:18.617243 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:18.617178    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-csrcb_d3a5a3f7-dc34-4f63-9fa9-5660515f8000/kube-rbac-proxy/0.log"
Sep 06 05:34:19.439875 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:19.439843    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-controller/0.log"
Sep 06 05:34:19.453003 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:19.452980    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/1.log"
Sep 06 05:34:19.453003 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:19.452997    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovn-acl-logging/0.log"
Sep 06 05:34:19.476633 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:19.476614    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/kube-rbac-proxy-node/0.log"
Sep 06 05:34:19.500749 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:19.500726    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/kube-rbac-proxy-ovn-metrics/0.log"
Sep 06 05:34:19.514759 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:19.514741    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/northd/0.log"
Sep 06 05:34:19.532457 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:19.532438    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/nbdb/0.log"
Sep 06 05:34:19.549353 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:19.549331    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/sbdb/0.log"
Sep 06 05:34:19.632880 ip-10-0-128-168 kubenswrapper[2580]: I0906 05:34:19.632853    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6gsk_7da02b5e-ca23-4376-9c84-7b1ff2713408/ovnkube-controller/0.log"