Sep 15 21:08:16.455251 ip-10-0-143-191 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 21:08:16.455263 ip-10-0-143-191 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 21:08:16.455273 ip-10-0-143-191 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 21:08:16.455457 ip-10-0-143-191 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 21:08:26.520999 ip-10-0-143-191 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 21:08:26.521014 ip-10-0-143-191 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 000613812ba44453bffdf73906f8f1bf --
Sep 15 21:10:09.923618 ip-10-0-143-191 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 21:10:10.328977 ip-10-0-143-191 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:10:10.328977 ip-10-0-143-191 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 21:10:10.328977 ip-10-0-143-191 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:10:10.328977 ip-10-0-143-191 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 21:10:10.328977 ip-10-0-143-191 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:10:10.331245 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.331181    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 21:10:10.336565 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336553    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:10.336565 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336565    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336568    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336573    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336576    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336580    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336583    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336586    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336589    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336591    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336594    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336597    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336600    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336603    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336605    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336608    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336610    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336613    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336615    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336622    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336625    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:10.336629 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336628    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336630    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336633    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336635    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336638    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336641    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336643    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336648    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336652    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336655    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336657    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336660    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336663    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336665    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336668    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336670    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336673    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336675    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336677    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336680    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:10.337083 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336682    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336685    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336689    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336691    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336693    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336696    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336698    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336701    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336703    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336706    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336709    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336711    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336714    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336716    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336719    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336722    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336724    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336727    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336729    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:10.337607 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336732    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336734    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336737    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336739    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336743    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336748    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336751    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336753    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336756    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336759    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336761    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336764    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336767    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336769    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336772    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336774    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336777    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336780    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336782    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:10.338056 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336785    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336787    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336790    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336792    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336794    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336797    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336799    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336802    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336806    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336810    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336813    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336816    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336819    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336823    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336825    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336828    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336830    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336833    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336835    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:10.338514 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.336840    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:10.339877 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339865    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:10.339877 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339878    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339882    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339885    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339888    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339891    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339894    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339896    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339900    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339905    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339907    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339910    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339913    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339915    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339918    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339920    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339923    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339925    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339929    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339932    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:10.339935 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339935    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339938    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339941    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339944    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339947    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339950    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339953    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339955    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339958    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339960    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339963    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339965    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339968    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339970    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339973    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339975    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339978    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339981    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339983    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339985    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:10.340401 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339988    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339990    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339994    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339996    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.339998    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340002    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340004    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340006    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340009    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340011    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340014    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340016    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340018    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340022    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340026    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340030    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340032    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340035    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340039    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:10.340863 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340042    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340044    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340047    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340050    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340053    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340056    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340059    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340062    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340065    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340067    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340070    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340073    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340076    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340079    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340081    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340084    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340087    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340089    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340092    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340094    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:10.341441 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340097    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340099    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340102    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340104    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340107    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340109    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340112    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340115    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340118    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340121    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340123    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340126    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340129    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340131    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340134    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340136    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340139    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340141    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340144    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340146    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:10.341933 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340517    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340522    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340525    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340527    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340530    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340532    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340535    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340538    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340541    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340543    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340546    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340548    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340552    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340554    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340557    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340559    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340562    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340565    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340568    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340571    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:10.342430 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340574    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340577    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340580    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340582    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340584    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340587    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340589    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340592    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340594    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340612    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340616    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340620    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340623    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340626    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340628    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340631    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340633    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340636    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340638    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340641    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:10.342898 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340643    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340647    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340649    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340652    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340654    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340657    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340659    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340661    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340664    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340667    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340670    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340674    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340677    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340681    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340685    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340688    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340691    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340694    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:10.343376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340697    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340701    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340704    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340707    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340710    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340712    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340715    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340717    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340720    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340722    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340725    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340727    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340730    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340732    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340734    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340737    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340740    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340743    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340745    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:10.343798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340748    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340750    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340753    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340756    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340759    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340761    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340764    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340766    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340769    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340772    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340774    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340776    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340779    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340782    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340784    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340787    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340789    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340791    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340794    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340796    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:10.344255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340799    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.340801    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340864    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340899    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340906    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340912    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340915    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340918    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340922    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340926    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340930    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340933    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340937    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340940    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340943    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340946    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340949    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340952    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340955    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340958    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340963    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340967    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340972    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340977    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 21:10:10.344747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340982    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340987    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340992    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.340996    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341001    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341006    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341011    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341015    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341021    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341026    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341031    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341036    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341041    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341045    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341050    2563 flags.go:64] FLAG: --enable-server="true"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341055    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341065    2563 flags.go:64] FLAG: --event-burst="100"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341072    2563 flags.go:64] FLAG: --event-qps="50"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341077    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341083    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341088    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341094    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341098    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341103    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341108    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 15 21:10:10.345289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341112    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341117    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341121    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341126    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341130    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341134    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341139    2563 flags.go:64] FLAG: --feature-gates=""
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341145    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341150    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341154    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341159    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341163    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341167    2563 flags.go:64] FLAG: --help="false"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341172    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341177    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341182    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341186    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341191    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341194    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341197    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341200    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341203    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341205    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 21:10:10.345972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341212    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341215    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341218    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341221    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341224    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341227    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341230    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341233    2563 flags.go:64] FLAG: --lock-file=""
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341236    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341239    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341242    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341247    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341250    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341252    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341255    2563 flags.go:64] FLAG: --logging-format="text"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341258    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341261    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341264    2563 flags.go:64] FLAG: --manifest-url=""
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341267    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341271    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341274    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341278    2563 flags.go:64] FLAG: --max-pods="110"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341281    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341284    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341286    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 21:10:10.346607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341289    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341292    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341295    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341298    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341309    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341312    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341317    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341320    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341323    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341328    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341331    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341334    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341337    2563 flags.go:64] FLAG: --port="10250"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341340    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341354    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00ad5742bb34654ff"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341358    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341360    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341363    2563 flags.go:64] FLAG: --register-node="true"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341366    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341370    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341373    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341376    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341379    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341385    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341388    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 21:10:10.347204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341391    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341394    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341397    2563 flags.go:64] FLAG: --runonce="false"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341400    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341403    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341406    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341409    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341411    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341414    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341417    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341420    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341423    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341426    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341429    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341434    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341437    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341440    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341443    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341448    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341451    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341455    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341459    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341462    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341465    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341468    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 21:10:10.347842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341471    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341473    2563 flags.go:64] FLAG: --v="2"
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341477    2563 flags.go:64] FLAG: --version="false"
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341481    2563 flags.go:64] FLAG: --vmodule=""
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341484    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.341488    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341611    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341615    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341619    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341622    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341625    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341627    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341631    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341634    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341637    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341646    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341661    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341665    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341668    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341671    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341674    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:10.348452 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341679    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341682    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341685    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341688    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341692    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341695    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341697    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341701    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341704    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341707    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341710    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341712    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341715    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341718    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341721    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341724    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341726    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341729    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341732    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:10.348952 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341734    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341737    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341741    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341744    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341747    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341749    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341753    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341755    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341758    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341761    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341763    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341765    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341768    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341773    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341776    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341778    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341781    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341783    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341786    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:10.349427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341789    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341792    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341795    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341797    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341800    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341802    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341805    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341807    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341810    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341812    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341815    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341817    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341820    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341822    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341825    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341828    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341830    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341833    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341835    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341838    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:10.349867 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341840    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341843    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341845    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341847    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341850    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341853    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341857    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341859    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341862    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341865    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341867    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341869    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341872    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341875    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341878    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341882    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341884    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341887    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341889    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341892    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:10.350380 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341894    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341897    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341900    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341902    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341904    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.341907    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.342695    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.349177    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.349193    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.350360    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.350371    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350429    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350435    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350439    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:10.350852 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350443    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350447    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350450    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350452    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350456    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350459    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350462    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350465    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350468    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350472    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350475    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350477    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350480    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350483    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350485    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350488    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350491    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350493    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350496    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350499    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:10.351188 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350502    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350505    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350508    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350510    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350513    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350516    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350526    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350529    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350532    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350535    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350538    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350541    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350543    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350546    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350549    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350551    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350554    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350556    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350559    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350561    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:10.351685 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350564    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350566    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350568    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350571    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350573    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350576    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350578    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350581    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350584    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350586    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350589    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350591    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350593    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350596    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350599    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350601    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350604    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350607    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350610    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350618    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:10.352149 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350620    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350623    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350625    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350628    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350630    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350633    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350635    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350638    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350640    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350643    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350645    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350648    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350650    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350653    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350655    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350658    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350660    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350663    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350665    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:10.352644 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350668    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350670    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350673    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350675    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350677    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350680    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350682    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350685    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350687    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350690    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350693    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350696    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350698    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350706    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350708    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350711    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:10.353090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350713    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.350718    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350855    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350860    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350863    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350866    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350869    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350872    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350875    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350877    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350880    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350883    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350885    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350888    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:10.353519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350891    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350893    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350896    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350898    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350901    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350903    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350906    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350909    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350911    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350913    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350916    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350918    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350922    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350926    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350929    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350932    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350940    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350943    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350946    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350949    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:10.353869 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350952    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350954    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350957    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350960    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350963    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350966    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350968    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350971    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350974    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350977    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350981    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350984    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350986    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350989    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350991    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350994    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.350997    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351000    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351002    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351005    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:10.354331 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351007    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351010    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351012    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351015    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351017    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351020    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351022    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351025    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351027    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351035    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351037    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351040    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351043    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351045    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351047    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351050    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351052    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351055    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351057    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351060    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:10.354829 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351062    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351065    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351067    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351070    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351072    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351075    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351077    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351080    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351082    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351084    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351087    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351089    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351092    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351095    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351097    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351099    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351103    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351106    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351109    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:10.355286 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351111    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351113    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351116    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351123    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351126    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351128    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351131    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:10.351133    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.351138    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.351259    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.353571    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.354492    2563 server.go:1019] "Starting client certificate rotation"
Sep 15 21:10:10.355806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.354594    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:10:10.356095 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.354622    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:10:10.359247 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.359230    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:10:10.360330 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.360313    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:10:10.371801 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.371786    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 15 21:10:10.377011 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.376995    2563 log.go:25] "Validated CRI v1 image API"
Sep 15 21:10:10.377869 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.377851    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 21:10:10.385402 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.385375    2563 fs.go:135] Filesystem UUIDs: map[0a188e87-3544-4fab-ad59-66fca2aa54fc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 aaded507-8700-4e4b-9629-0bc1117dbfa1:/dev/nvme0n1p3]
Sep 15 21:10:10.385465 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.385401    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 21:10:10.387613 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.387597    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:10:10.390502 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.390401    2563 manager.go:216] Machine: {Timestamp:2026-09-15 21:10:10.389093728 +0000 UTC m=+0.352151100 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199542 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b0ec069222ac05d773ee06a12ea05 SystemUUID:ec2b0ec0-6922-2ac0-5d77-3ee06a12ea05 BootID:00061381-2ba4-4453-bffd-f73906f8f1bf Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:c3:1d:4b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:c3:1d:4b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:e0:c2:3d:c9:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 21:10:10.390502 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.390496    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 21:10:10.390605 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.390563    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 21:10:10.393788 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.393759    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 21:10:10.393935 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.393791    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 21:10:10.394530 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.394520    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 21:10:10.394560 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.394533    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 21:10:10.394560 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.394540    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:10:10.395166 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.395157    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:10:10.395987 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.395978    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 21:10:10.399680 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.399668    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:10:10.399779 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.399770    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 21:10:10.399808 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.399801    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 21:10:10.399837 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.399811    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 21:10:10.399837 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.399833    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 21:10:10.399889 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.399847    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 15 21:10:10.399889 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.399857    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 21:10:10.400288 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.400269    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:10:10.400393 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.400294    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:10:10.403750 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.403703    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 21:10:10.405730 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.405716    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 21:10:10.405795 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.405745    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 21:10:10.407536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407524    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407540    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407546    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407551    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407560    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407565    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407577    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407582    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407588    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407594    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407601    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 21:10:10.407635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.407606    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 21:10:10.408616 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.408602    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 21:10:10.408616 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.408614    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 21:10:10.408784 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.408760    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 21:10:10.408861 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.408842    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 21:10:10.411898 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.411886    2563 server.go:1268] "Started kubelet"
Sep 15 21:10:10.411977 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.411945    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 21:10:10.412432 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.412041    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 21:10:10.412432 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.412108    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 21:10:10.412698 ip-10-0-143-191 systemd[1]: Started Kubernetes Kubelet.
Sep 15 21:10:10.413103 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.413089    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 21:10:10.413407 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.413390    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 21:10:10.414916 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.414900    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5pmv4"
Sep 15 21:10:10.417051 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.416804    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 21:10:10.418703 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.418681    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-191.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 21:10:10.419646 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.418754    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-191.ec2.internal.18d59ac7eccff615  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-191.ec2.internal,UID:ip-10-0-143-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-191.ec2.internal,},FirstTimestamp:2026-09-15 21:10:10.411861525 +0000 UTC m=+0.374918896,LastTimestamp:2026-09-15 21:10:10.411861525 +0000 UTC m=+0.374918896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-191.ec2.internal,}"
Sep 15 21:10:10.420776 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.420762    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5pmv4"
Sep 15 21:10:10.422087 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.422067    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 21:10:10.422664 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.422647    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 21:10:10.422770 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.422753    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:10.423203 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423186    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 21:10:10.423283 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423227    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 21:10:10.423283 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423241    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 21:10:10.423386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423367    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 21:10:10.423386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423377    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 21:10:10.423624 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423609    2563 factory.go:55] Registering systemd factory
Sep 15 21:10:10.423624 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423626    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 15 21:10:10.423797 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423786    2563 factory.go:153] Registering CRI-O factory
Sep 15 21:10:10.423849 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423801    2563 factory.go:223] Registration of the crio container factory successfully
Sep 15 21:10:10.423901 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423857    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 21:10:10.423901 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423875    2563 factory.go:103] Registering Raw factory
Sep 15 21:10:10.423901 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.423890    2563 manager.go:1196] Started watching for new ooms in manager
Sep 15 21:10:10.424283 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.424270    2563 manager.go:349] Starting recovery of all containers
Sep 15 21:10:10.424666 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.424625    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 21:10:10.424763 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.424740    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 21:10:10.436237 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.436220    2563 manager.go:354] Recovery completed
Sep 15 21:10:10.441519 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.441505    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:10.443914 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.443896    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:10.443972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.443926    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:10.443972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.443936    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:10.444460 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.444440    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 21:10:10.444460 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.444452    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:10.444602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.444455    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 21:10:10.444642 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.444632    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:10:10.445131 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.445115    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:10.445191 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.445141    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:10.445191 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.445150    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:10.446643 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.446630    2563 policy_none.go:49] "None policy: Start"
Sep 15 21:10:10.446704 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.446645    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 21:10:10.446704 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.446655    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 21:10:10.448882 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.448869    2563 policy_none.go:47] "Start"
Sep 15 21:10:10.485975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.485961    2563 manager.go:337] "Starting Device Plugin manager"
Sep 15 21:10:10.504764 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.485994    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 21:10:10.504764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.486007    2563 server.go:87] "Starting device plugin registration server"
Sep 15 21:10:10.504764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.486273    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 21:10:10.504764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.486294    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 21:10:10.504764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.486396    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 21:10:10.504764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.486473    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 21:10:10.504764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.486482    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 21:10:10.504764 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.486966    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 21:10:10.504764 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.487014    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:10.518008 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.517988    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 21:10:10.519063 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.519045    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 21:10:10.519143 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.519098    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 21:10:10.519143 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.519120    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 21:10:10.519238 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.519187    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 21:10:10.521781 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.521760    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:10.587201 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.587156    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:10.587895 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.587871    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:10.587895 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.587896    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:10.588031 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.587908    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:10.588031 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.587934    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.597465 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.597441    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.597465 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.597460    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-191.ec2.internal\": node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:10.608807 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.608778    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:10.619793 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.619775    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal"]
Sep 15 21:10:10.619850 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.619839    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:10.620492 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.620480    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:10.620542 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.620502    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:10.620542 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.620513    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:10.622490 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.622478    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:10.622580 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.622567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.622614 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.622598    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:10.623125 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.623112    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:10.623197 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.623130    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:10.623197 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.623139    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:10.623197 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.623115    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:10.623197 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.623169    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:10.623197 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.623181    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:10.624174 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.624159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/785230b8847b3f5457cd788cddf1108f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal\" (UID: \"785230b8847b3f5457cd788cddf1108f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.624225 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.624187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/785230b8847b3f5457cd788cddf1108f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal\" (UID: \"785230b8847b3f5457cd788cddf1108f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.624225 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.624207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfa4cb4183b054641aa8c971a669467e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-191.ec2.internal\" (UID: \"bfa4cb4183b054641aa8c971a669467e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.625127 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.625109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.625216 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.625134    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:10.625728 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.625711    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:10.625792 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.625737    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:10.625792 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.625750    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:10.646900 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.646879    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-191.ec2.internal\" not found" node="ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.651006 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.650992    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-191.ec2.internal\" not found" node="ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.709390 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.709373    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:10.724334 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.724314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/785230b8847b3f5457cd788cddf1108f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal\" (UID: \"785230b8847b3f5457cd788cddf1108f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.724423 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.724364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/785230b8847b3f5457cd788cddf1108f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal\" (UID: \"785230b8847b3f5457cd788cddf1108f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.724423 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.724392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfa4cb4183b054641aa8c971a669467e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-191.ec2.internal\" (UID: \"bfa4cb4183b054641aa8c971a669467e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.724423 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.724400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/785230b8847b3f5457cd788cddf1108f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal\" (UID: \"785230b8847b3f5457cd788cddf1108f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.724423 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.724413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/785230b8847b3f5457cd788cddf1108f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal\" (UID: \"785230b8847b3f5457cd788cddf1108f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.724549 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.724424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfa4cb4183b054641aa8c971a669467e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-191.ec2.internal\" (UID: \"bfa4cb4183b054641aa8c971a669467e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.809654 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.809631    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:10.910306 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:10.910262    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:10.949712 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.949688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:10.953913 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:10.953900    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:11.011273 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:11.011251    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:11.111742 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:11.111717    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:11.212222 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:11.212166    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:11.312603 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:11.312582    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:11.353960 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.353945    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 21:10:11.354395 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.354048    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 21:10:11.413379 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:11.413362    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:11.424531 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.424507    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 21:05:10 +0000 UTC" deadline="2028-03-02 00:31:48.239442111 +0000 UTC"
Sep 15 21:10:11.424531 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.424529    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12795h21m36.81491529s"
Sep 15 21:10:11.513673 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:11.513652    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:11.517185 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:11.517150    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa4cb4183b054641aa8c971a669467e.slice/crio-4200cfde86032864ab282ec1ddf1179cc0490c46c7230248e1d930535037d13f WatchSource:0}: Error finding container 4200cfde86032864ab282ec1ddf1179cc0490c46c7230248e1d930535037d13f: Status 404 returned error can't find the container with id 4200cfde86032864ab282ec1ddf1179cc0490c46c7230248e1d930535037d13f
Sep 15 21:10:11.518199 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:11.518172    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785230b8847b3f5457cd788cddf1108f.slice/crio-447b9b22a14275ca1a4c738c25255743f3a3e35501f5d64dcbc99ba948b63976 WatchSource:0}: Error finding container 447b9b22a14275ca1a4c738c25255743f3a3e35501f5d64dcbc99ba948b63976: Status 404 returned error can't find the container with id 447b9b22a14275ca1a4c738c25255743f3a3e35501f5d64dcbc99ba948b63976
Sep 15 21:10:11.522242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.522229    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:10:11.522942 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.522902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal" event={"ID":"785230b8847b3f5457cd788cddf1108f","Type":"ContainerStarted","Data":"447b9b22a14275ca1a4c738c25255743f3a3e35501f5d64dcbc99ba948b63976"}
Sep 15 21:10:11.523932 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.523913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal" event={"ID":"bfa4cb4183b054641aa8c971a669467e","Type":"ContainerStarted","Data":"4200cfde86032864ab282ec1ddf1179cc0490c46c7230248e1d930535037d13f"}
Sep 15 21:10:11.614100 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:11.614070    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:11.714446 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:11.714424    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:11.735834 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.735816    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:11.814624 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:11.814583    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-191.ec2.internal\" not found"
Sep 15 21:10:11.904483 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.904469    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:11.923592 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.923574    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:11.924822 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.924800    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:11.933833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.933812    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:10:11.934683 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.934671    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal"
Sep 15 21:10:11.944859 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:11.944843    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:10:12.401009 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.400979    2563 apiserver.go:52] "Watching apiserver"
Sep 15 21:10:12.406357 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.406317    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 21:10:12.406759 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.406734    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-2mthm","kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal","openshift-multus/multus-additional-cni-plugins-rc8jl","openshift-network-diagnostics/network-check-target-z477j","openshift-network-operator/iptables-alerter-kdr47","openshift-ovn-kubernetes/ovnkube-node-brhtn","kube-system/konnectivity-agent-b5ndc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck","openshift-cluster-node-tuning-operator/tuned-ltn26","openshift-dns/node-resolver-sds6c","openshift-image-registry/node-ca-hcnhm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal","openshift-multus/multus-wqnjj","openshift-multus/network-metrics-daemon-gp2b5"]
Sep 15 21:10:12.409153 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.409135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:12.411292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.411261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.411976 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.411621    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 21:10:12.411976 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.411763    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8wf4r\""
Sep 15 21:10:12.411976 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.411800    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 21:10:12.413729 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.413711    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.414453 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.414435    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 21:10:12.414566 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.414547    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 21:10:12.414624 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.414436    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-plmfd\""
Sep 15 21:10:12.414776 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.414760    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 21:10:12.414899 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.414765    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 21:10:12.416779 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.415680    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 21:10:12.416779 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.415920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r5t9p\""
Sep 15 21:10:12.416779 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.416428    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 21:10:12.416779 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.416706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:12.417008 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.416798    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:12.418197 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.418003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.421744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.420360    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qnnv5\""
Sep 15 21:10:12.421744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.420431    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 21:10:12.421744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.420494    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 21:10:12.421744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.420550    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:10:12.421744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.420798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.422737 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.422717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 21:10:12.423385 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.422877    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 21:10:12.423385 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.422986    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 21:10:12.423385 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.423185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 21:10:12.423385 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.423198    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 21:10:12.423385 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.423265    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 21:10:12.423385 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.423332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.423854 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.423837    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v5kc5\""
Sep 15 21:10:12.424124 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.424102    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 21:10:12.425368 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.425339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:12.425460 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.425417    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:12.425522 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.425499    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lv5gd\""
Sep 15 21:10:12.425584 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.425567    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:10:12.425738 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.425709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 21:10:12.428389 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.428267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:12.428389 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.428329    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:12.432001 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.431704    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d76ae1c9-7503-48a0-9eed-5b3539718a70-agent-certs\") pod \"konnectivity-agent-b5ndc\" (UID: \"d76ae1c9-7503-48a0-9eed-5b3539718a70\") " pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-run-ovn-kubernetes\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-run\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e74e7fd1-2672-4c85-b798-a3f036fb7681-kubelet-config\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-cni-dir\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-hostroot\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdd9\" (UniqueName: \"kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9\") pod \"network-check-target-z477j\" (UID: \"4832498c-7ee0-4464-aae8-919202c79bdd\") " pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-kubelet\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-node-log\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbc240ac-5432-43b5-ba3d-843e08df0eb9-ovn-node-metrics-cert\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-tuned\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-cni-binary-copy\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-var-lib-cni-multus\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-cni-netd\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.432867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-var-lib-kubelet\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dc6096-8902-4327-b266-0d73cb2f676d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-run-systemd\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-etc-openvswitch\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-sysconfig\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.432982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e74e7fd1-2672-4c85-b798-a3f036fb7681-dbus\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-system-cni-dir\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dc6096-8902-4327-b266-0d73cb2f676d-system-cni-dir\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30f82454-0b7e-4f7b-b186-b44771628f4b-host-slash\") pod \"iptables-alerter-kdr47\" (UID: \"30f82454-0b7e-4f7b-b186-b44771628f4b\") " pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-run-ovn\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-run-netns\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-etc-kubernetes\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/30f82454-0b7e-4f7b-b186-b44771628f4b-iptables-alerter-script\") pod \"iptables-alerter-kdr47\" (UID: \"30f82454-0b7e-4f7b-b186-b44771628f4b\") " pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-run-netns\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dc6096-8902-4327-b266-0d73cb2f676d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.433829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbc240ac-5432-43b5-ba3d-843e08df0eb9-ovnkube-script-lib\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82m7\" (UniqueName: \"kubernetes.io/projected/2bd798c0-c26c-4ac1-9ae0-e5023908313b-kube-api-access-h82m7\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-cnibin\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-kubernetes\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm8x6\" (UniqueName: \"kubernetes.io/projected/51dc6096-8902-4327-b266-0d73cb2f676d-kube-api-access-xm8x6\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-sysctl-conf\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433832    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-host\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-socket-dir-parent\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-run-k8s-cni-cncf-io\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434086    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2sx\" (UniqueName: \"kubernetes.io/projected/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-kube-api-access-kx2sx\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.433986    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-klvt7\""
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dc6096-8902-4327-b266-0d73cb2f676d-cnibin\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-slash\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-var-lib-openvswitch\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d76ae1c9-7503-48a0-9eed-5b3539718a70-konnectivity-ca\") pod \"konnectivity-agent-b5ndc\" (UID: \"d76ae1c9-7503-48a0-9eed-5b3539718a70\") " pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dc6096-8902-4327-b266-0d73cb2f676d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.434641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt4kz\" (UniqueName: \"kubernetes.io/projected/30f82454-0b7e-4f7b-b186-b44771628f4b-kube-api-access-mt4kz\") pod \"iptables-alerter-kdr47\" (UID: \"30f82454-0b7e-4f7b-b186-b44771628f4b\") " pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-systemd-units\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-sysctl-d\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-var-lib-kubelet\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-daemon-config\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-log-socket\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbc240ac-5432-43b5-ba3d-843e08df0eb9-env-overrides\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v6sm\" (UniqueName: \"kubernetes.io/projected/cbc240ac-5432-43b5-ba3d-843e08df0eb9-kube-api-access-2v6sm\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-lib-modules\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-run-openvswitch\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbc240ac-5432-43b5-ba3d-843e08df0eb9-ovnkube-config\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-sys\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bd798c0-c26c-4ac1-9ae0-e5023908313b-tmp\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-conf-dir\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-run-multus-certs\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.434985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dc6096-8902-4327-b266-0d73cb2f676d-cni-binary-copy\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.435529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.435008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-cni-bin\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.436321 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.435027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-modprobe-d\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.436321 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.435048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-systemd\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.436321 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.435073    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-var-lib-cni-bin\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.436321 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.435298    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:10:12.436844 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.436809    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sw57n\""
Sep 15 21:10:12.436943 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.436852    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 21:10:12.437029 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.436966    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 21:10:12.437129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.437110    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 21:10:12.437279 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.437257    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.438958 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.438930    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 21:10:12.439542 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.439328    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 21:10:12.439542 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.439486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2v89c\""
Sep 15 21:10:12.439542 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.439521    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 21:10:12.451852 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.451834    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:12.452892 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.452872    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bpdzm"
Sep 15 21:10:12.459764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.459750    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bpdzm"
Sep 15 21:10:12.524132 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.524112    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 21:10:12.535361 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-run-openvswitch\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.535454 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbc240ac-5432-43b5-ba3d-843e08df0eb9-ovnkube-config\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.535518 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-run-openvswitch\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.535518 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-sys\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.535616 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bd798c0-c26c-4ac1-9ae0-e5023908313b-tmp\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.535616 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgkh\" (UniqueName: \"kubernetes.io/projected/8060115d-0d03-495c-bd01-c0a4315c501f-kube-api-access-tjgkh\") pod \"node-resolver-sds6c\" (UID: \"8060115d-0d03-495c-bd01-c0a4315c501f\") " pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.535616 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-conf-dir\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.535616 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-sys\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.535794 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-registration-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.535794 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-run-multus-certs\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.535794 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dc6096-8902-4327-b266-0d73cb2f676d-cni-binary-copy\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.535794 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-cni-bin\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.535794 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-conf-dir\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.535794 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-modprobe-d\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.536047 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-modprobe-d\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.536047 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-systemd\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.536047 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.535940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-sys-fs\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.536047 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-var-lib-cni-bin\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.536232 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:12.536232 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbc240ac-5432-43b5-ba3d-843e08df0eb9-ovnkube-config\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.536232 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d76ae1c9-7503-48a0-9eed-5b3539718a70-agent-certs\") pod \"konnectivity-agent-b5ndc\" (UID: \"d76ae1c9-7503-48a0-9eed-5b3539718a70\") " pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:12.536232 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536111    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 21:10:12.536232 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-systemd\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.536457 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-run-ovn-kubernetes\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.536457 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-run\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.536457 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e74e7fd1-2672-4c85-b798-a3f036fb7681-kubelet-config\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:12.536457 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-cni-dir\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.536457 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-hostroot\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.536681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdd9\" (UniqueName: \"kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9\") pod \"network-check-target-z477j\" (UID: \"4832498c-7ee0-4464-aae8-919202c79bdd\") " pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:12.536681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-kubelet\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.536681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-node-log\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.536681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbc240ac-5432-43b5-ba3d-843e08df0eb9-ovn-node-metrics-cert\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.536681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-tuned\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.536681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-cni-binary-copy\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.536681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dc6096-8902-4327-b266-0d73cb2f676d-cni-binary-copy\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.536681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-var-lib-cni-multus\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.537079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-cni-netd\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.537079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8m8k\" (UniqueName: \"kubernetes.io/projected/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-kube-api-access-t8m8k\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:12.537079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-node-log\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.537079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8060115d-0d03-495c-bd01-c0a4315c501f-tmp-dir\") pod \"node-resolver-sds6c\" (UID: \"8060115d-0d03-495c-bd01-c0a4315c501f\") " pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.537079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61e8e166-9aa3-4f53-b9d2-564ce990235c-host\") pod \"node-ca-hcnhm\" (UID: \"61e8e166-9aa3-4f53-b9d2-564ce990235c\") " pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.537079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-run\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.537079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-var-lib-kubelet\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.537079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-cni-dir\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.537079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.536985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dc6096-8902-4327-b266-0d73cb2f676d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.537079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-hostroot\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-run-systemd\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-etc-openvswitch\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-sysconfig\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e74e7fd1-2672-4c85-b798-a3f036fb7681-dbus\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-system-cni-dir\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dc6096-8902-4327-b266-0d73cb2f676d-system-cni-dir\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30f82454-0b7e-4f7b-b186-b44771628f4b-host-slash\") pod \"iptables-alerter-kdr47\" (UID: \"30f82454-0b7e-4f7b-b186-b44771628f4b\") " pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-run-ovn\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-etc-selinux\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-run-netns\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.537554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-etc-kubernetes\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.538513 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.538440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dc6096-8902-4327-b266-0d73cb2f676d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.538657 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.538624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/30f82454-0b7e-4f7b-b186-b44771628f4b-iptables-alerter-script\") pod \"iptables-alerter-kdr47\" (UID: \"30f82454-0b7e-4f7b-b186-b44771628f4b\") " pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.538740 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.538710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-run-netns\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.538793 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.538762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.538881 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.538864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dc6096-8902-4327-b266-0d73cb2f676d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.538932 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.538898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbc240ac-5432-43b5-ba3d-843e08df0eb9-ovnkube-script-lib\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.538981 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.538930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82m7\" (UniqueName: \"kubernetes.io/projected/2bd798c0-c26c-4ac1-9ae0-e5023908313b-kube-api-access-h82m7\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.538981 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.538963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.539067 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.538994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-socket-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.539067 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-cnibin\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.539159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-kubernetes\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.539159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh46g\" (UniqueName: \"kubernetes.io/projected/61e8e166-9aa3-4f53-b9d2-564ce990235c-kube-api-access-dh46g\") pod \"node-ca-hcnhm\" (UID: \"61e8e166-9aa3-4f53-b9d2-564ce990235c\") " pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.539242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm8x6\" (UniqueName: \"kubernetes.io/projected/51dc6096-8902-4327-b266-0d73cb2f676d-kube-api-access-xm8x6\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.539242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-sysctl-conf\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.539400 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-host\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.539400 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:12.539400 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8060115d-0d03-495c-bd01-c0a4315c501f-hosts-file\") pod \"node-resolver-sds6c\" (UID: \"8060115d-0d03-495c-bd01-c0a4315c501f\") " pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.539536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-socket-dir-parent\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.539536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-run-k8s-cni-cncf-io\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.539536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2sx\" (UniqueName: \"kubernetes.io/projected/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-kube-api-access-kx2sx\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.539681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.539542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dc6096-8902-4327-b266-0d73cb2f676d-cnibin\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.540275 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-slash\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.540360 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-var-lib-openvswitch\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.540360 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-device-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.540439 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc24h\" (UniqueName: \"kubernetes.io/projected/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-kube-api-access-lc24h\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.540439 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bd798c0-c26c-4ac1-9ae0-e5023908313b-tmp\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.540439 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d76ae1c9-7503-48a0-9eed-5b3539718a70-konnectivity-ca\") pod \"konnectivity-agent-b5ndc\" (UID: \"d76ae1c9-7503-48a0-9eed-5b3539718a70\") " pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:12.540523 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dc6096-8902-4327-b266-0d73cb2f676d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.540523 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt4kz\" (UniqueName: \"kubernetes.io/projected/30f82454-0b7e-4f7b-b186-b44771628f4b-kube-api-access-mt4kz\") pod \"iptables-alerter-kdr47\" (UID: \"30f82454-0b7e-4f7b-b186-b44771628f4b\") " pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.540582 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-systemd-units\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.540612 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-sysctl-d\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.540643 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-var-lib-kubelet\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.540671 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61e8e166-9aa3-4f53-b9d2-564ce990235c-serviceca\") pod \"node-ca-hcnhm\" (UID: \"61e8e166-9aa3-4f53-b9d2-564ce990235c\") " pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.540709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-daemon-config\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.540742 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-log-socket\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.540783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbc240ac-5432-43b5-ba3d-843e08df0eb9-env-overrides\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.540830 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6sm\" (UniqueName: \"kubernetes.io/projected/cbc240ac-5432-43b5-ba3d-843e08df0eb9-kube-api-access-2v6sm\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.540923 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-lib-modules\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.540984 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.540966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d76ae1c9-7503-48a0-9eed-5b3539718a70-agent-certs\") pod \"konnectivity-agent-b5ndc\" (UID: \"d76ae1c9-7503-48a0-9eed-5b3539718a70\") " pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-var-lib-kubelet\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-var-lib-cni-multus\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-run-ovn-kubernetes\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-lib-modules\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-var-lib-cni-bin\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dc6096-8902-4327-b266-0d73cb2f676d-cnibin\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbc240ac-5432-43b5-ba3d-843e08df0eb9-ovn-node-metrics-cert\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541295    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.541310    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-etc-kubernetes\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.541415    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret podName:e74e7fd1-2672-4c85-b798-a3f036fb7681 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:13.041371693 +0000 UTC m=+3.004429076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret") pod "global-pull-secret-syncer-2mthm" (UID: "e74e7fd1-2672-4c85-b798-a3f036fb7681") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-socket-dir-parent\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-run-k8s-cni-cncf-io\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-slash\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-var-lib-openvswitch\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-etc-openvswitch\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.542386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-sysconfig\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/30f82454-0b7e-4f7b-b186-b44771628f4b-iptables-alerter-script\") pod \"iptables-alerter-kdr47\" (UID: \"30f82454-0b7e-4f7b-b186-b44771628f4b\") " pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dc6096-8902-4327-b266-0d73cb2f676d-system-cni-dir\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.541990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30f82454-0b7e-4f7b-b186-b44771628f4b-host-slash\") pod \"iptables-alerter-kdr47\" (UID: \"30f82454-0b7e-4f7b-b186-b44771628f4b\") " pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.537391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-cni-binary-copy\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-run-ovn\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e74e7fd1-2672-4c85-b798-a3f036fb7681-dbus\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-run-netns\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542260    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-sysctl-d\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-run-systemd\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-kubernetes\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-var-lib-kubelet\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-host\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-system-cni-dir\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-sysctl-conf\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-multus-daemon-config\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-cnibin\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-log-socket\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.542921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.542892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-kubelet\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.543427 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d76ae1c9-7503-48a0-9eed-5b3539718a70-konnectivity-ca\") pod \"konnectivity-agent-b5ndc\" (UID: \"d76ae1c9-7503-48a0-9eed-5b3539718a70\") " pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:12.543427 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e74e7fd1-2672-4c85-b798-a3f036fb7681-kubelet-config\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:12.543427 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-run-netns\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.543427 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dc6096-8902-4327-b266-0d73cb2f676d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.543536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbc240ac-5432-43b5-ba3d-843e08df0eb9-ovnkube-script-lib\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.543536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-cni-netd\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.544278 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbc240ac-5432-43b5-ba3d-843e08df0eb9-env-overrides\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.544278 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-host-cni-bin\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.544278 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-host-run-multus-certs\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.544278 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbc240ac-5432-43b5-ba3d-843e08df0eb9-systemd-units\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.544278 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.543989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dc6096-8902-4327-b266-0d73cb2f676d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.545765 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.545730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2bd798c0-c26c-4ac1-9ae0-e5023908313b-etc-tuned\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.552118 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.551929    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:12.552118 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.551949    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:12.552118 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.551963    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfdd9 for pod openshift-network-diagnostics/network-check-target-z477j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:12.552118 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.552014    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9 podName:4832498c-7ee0-4464-aae8-919202c79bdd nodeName:}" failed. No retries permitted until 2026-09-15 21:10:13.051997141 +0000 UTC m=+3.015054523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cfdd9" (UniqueName: "kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9") pod "network-check-target-z477j" (UID: "4832498c-7ee0-4464-aae8-919202c79bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:12.553199 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.553141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2sx\" (UniqueName: \"kubernetes.io/projected/e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6-kube-api-access-kx2sx\") pod \"multus-wqnjj\" (UID: \"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6\") " pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.554675 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.554630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v6sm\" (UniqueName: \"kubernetes.io/projected/cbc240ac-5432-43b5-ba3d-843e08df0eb9-kube-api-access-2v6sm\") pod \"ovnkube-node-brhtn\" (UID: \"cbc240ac-5432-43b5-ba3d-843e08df0eb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.555077 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.555055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt4kz\" (UniqueName: \"kubernetes.io/projected/30f82454-0b7e-4f7b-b186-b44771628f4b-kube-api-access-mt4kz\") pod \"iptables-alerter-kdr47\" (UID: \"30f82454-0b7e-4f7b-b186-b44771628f4b\") " pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.555340 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.555315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82m7\" (UniqueName: \"kubernetes.io/projected/2bd798c0-c26c-4ac1-9ae0-e5023908313b-kube-api-access-h82m7\") pod \"tuned-ltn26\" (UID: \"2bd798c0-c26c-4ac1-9ae0-e5023908313b\") " pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.557189 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.557171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm8x6\" (UniqueName: \"kubernetes.io/projected/51dc6096-8902-4327-b266-0d73cb2f676d-kube-api-access-xm8x6\") pod \"multus-additional-cni-plugins-rc8jl\" (UID: \"51dc6096-8902-4327-b266-0d73cb2f676d\") " pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.641545 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-sys-fs\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.641679 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8m8k\" (UniqueName: \"kubernetes.io/projected/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-kube-api-access-t8m8k\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:12.641679 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-sys-fs\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.641774 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8060115d-0d03-495c-bd01-c0a4315c501f-tmp-dir\") pod \"node-resolver-sds6c\" (UID: \"8060115d-0d03-495c-bd01-c0a4315c501f\") " pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.641774 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61e8e166-9aa3-4f53-b9d2-564ce990235c-host\") pod \"node-ca-hcnhm\" (UID: \"61e8e166-9aa3-4f53-b9d2-564ce990235c\") " pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.641869 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-etc-selinux\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.641869 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.641869 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-socket-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.641869 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61e8e166-9aa3-4f53-b9d2-564ce990235c-host\") pod \"node-ca-hcnhm\" (UID: \"61e8e166-9aa3-4f53-b9d2-564ce990235c\") " pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.641869 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh46g\" (UniqueName: \"kubernetes.io/projected/61e8e166-9aa3-4f53-b9d2-564ce990235c-kube-api-access-dh46g\") pod \"node-ca-hcnhm\" (UID: \"61e8e166-9aa3-4f53-b9d2-564ce990235c\") " pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-etc-selinux\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8060115d-0d03-495c-bd01-c0a4315c501f-hosts-file\") pod \"node-resolver-sds6c\" (UID: \"8060115d-0d03-495c-bd01-c0a4315c501f\") " pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-device-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.641992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc24h\" (UniqueName: \"kubernetes.io/projected/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-kube-api-access-lc24h\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.642029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8060115d-0d03-495c-bd01-c0a4315c501f-hosts-file\") pod \"node-resolver-sds6c\" (UID: \"8060115d-0d03-495c-bd01-c0a4315c501f\") " pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.642033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61e8e166-9aa3-4f53-b9d2-564ce990235c-serviceca\") pod \"node-ca-hcnhm\" (UID: \"61e8e166-9aa3-4f53-b9d2-564ce990235c\") " pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.641988    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.642072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-device-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.642081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgkh\" (UniqueName: \"kubernetes.io/projected/8060115d-0d03-495c-bd01-c0a4315c501f-kube-api-access-tjgkh\") pod \"node-resolver-sds6c\" (UID: \"8060115d-0d03-495c-bd01-c0a4315c501f\") " pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.642111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.642107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-registration-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.642684 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:12.642122    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs podName:a25ac072-b1cb-4739-a1e7-f0ed49ddf881 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:13.142103724 +0000 UTC m=+3.105161085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs") pod "network-metrics-daemon-gp2b5" (UID: "a25ac072-b1cb-4739-a1e7-f0ed49ddf881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:12.642684 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.642145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-socket-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.642684 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.642155    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-registration-dir\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.642684 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.642029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8060115d-0d03-495c-bd01-c0a4315c501f-tmp-dir\") pod \"node-resolver-sds6c\" (UID: \"8060115d-0d03-495c-bd01-c0a4315c501f\") " pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.642684 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.642495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61e8e166-9aa3-4f53-b9d2-564ce990235c-serviceca\") pod \"node-ca-hcnhm\" (UID: \"61e8e166-9aa3-4f53-b9d2-564ce990235c\") " pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.652603 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.652547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh46g\" (UniqueName: \"kubernetes.io/projected/61e8e166-9aa3-4f53-b9d2-564ce990235c-kube-api-access-dh46g\") pod \"node-ca-hcnhm\" (UID: \"61e8e166-9aa3-4f53-b9d2-564ce990235c\") " pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.652895 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.652869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgkh\" (UniqueName: \"kubernetes.io/projected/8060115d-0d03-495c-bd01-c0a4315c501f-kube-api-access-tjgkh\") pod \"node-resolver-sds6c\" (UID: \"8060115d-0d03-495c-bd01-c0a4315c501f\") " pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.653356 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.653320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8m8k\" (UniqueName: \"kubernetes.io/projected/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-kube-api-access-t8m8k\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:12.654029 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.654007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc24h\" (UniqueName: \"kubernetes.io/projected/18b009b6-fdf0-47c4-bff1-de5e0c1db03e-kube-api-access-lc24h\") pod \"aws-ebs-csi-driver-node-blfck\" (UID: \"18b009b6-fdf0-47c4-bff1-de5e0c1db03e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:12.723776 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.723753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:12.736910 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.736876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wqnjj"
Sep 15 21:10:12.744587 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.744554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rc8jl"
Sep 15 21:10:12.751186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.751169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kdr47"
Sep 15 21:10:12.757607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.757589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:12.766088 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.766072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ltn26"
Sep 15 21:10:12.771488 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.771473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sds6c"
Sep 15 21:10:12.777736 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.777719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hcnhm"
Sep 15 21:10:12.783239 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:12.783221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck"
Sep 15 21:10:13.043983 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.043900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:13.044131 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:13.044060    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:13.044131 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:13.044130    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret podName:e74e7fd1-2672-4c85-b798-a3f036fb7681 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:14.04411133 +0000 UTC m=+4.007168695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret") pod "global-pull-secret-syncer-2mthm" (UID: "e74e7fd1-2672-4c85-b798-a3f036fb7681") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:13.145146 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.145115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdd9\" (UniqueName: \"kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9\") pod \"network-check-target-z477j\" (UID: \"4832498c-7ee0-4464-aae8-919202c79bdd\") " pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:13.145302 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.145163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:13.145302 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:13.145270    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:13.145302 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:13.145274    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:13.145302 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:13.145292    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:13.145302 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:13.145300    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfdd9 for pod openshift-network-diagnostics/network-check-target-z477j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:13.145500 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:13.145329    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs podName:a25ac072-b1cb-4739-a1e7-f0ed49ddf881 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:14.145310891 +0000 UTC m=+4.108368264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs") pod "network-metrics-daemon-gp2b5" (UID: "a25ac072-b1cb-4739-a1e7-f0ed49ddf881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:13.145500 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:13.145363    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9 podName:4832498c-7ee0-4464-aae8-919202c79bdd nodeName:}" failed. No retries permitted until 2026-09-15 21:10:14.145337854 +0000 UTC m=+4.108395215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfdd9" (UniqueName: "kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9") pod "network-check-target-z477j" (UID: "4832498c-7ee0-4464-aae8-919202c79bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:13.410720 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:13.410693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a45cb7_2e0d_4e7a_b855_3d40b108cdc6.slice/crio-166ac5b04f612b37e73f94c5f78239a3155e70948a0b861ba19de47a29f34e67 WatchSource:0}: Error finding container 166ac5b04f612b37e73f94c5f78239a3155e70948a0b861ba19de47a29f34e67: Status 404 returned error can't find the container with id 166ac5b04f612b37e73f94c5f78239a3155e70948a0b861ba19de47a29f34e67
Sep 15 21:10:13.415599 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:13.414933    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f82454_0b7e_4f7b_b186_b44771628f4b.slice/crio-8c58ac2f04e75ccad8ef4f8d509d1d2641f35764906e9fca785493db93f7d1e0 WatchSource:0}: Error finding container 8c58ac2f04e75ccad8ef4f8d509d1d2641f35764906e9fca785493db93f7d1e0: Status 404 returned error can't find the container with id 8c58ac2f04e75ccad8ef4f8d509d1d2641f35764906e9fca785493db93f7d1e0
Sep 15 21:10:13.417519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:13.417493    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e8e166_9aa3_4f53_b9d2_564ce990235c.slice/crio-1a3d2658520eae2a3c4cdfd89823183fbd908658dfe492371c39ab8f8deec220 WatchSource:0}: Error finding container 1a3d2658520eae2a3c4cdfd89823183fbd908658dfe492371c39ab8f8deec220: Status 404 returned error can't find the container with id 1a3d2658520eae2a3c4cdfd89823183fbd908658dfe492371c39ab8f8deec220
Sep 15 21:10:13.420072 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:13.419900    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76ae1c9_7503_48a0_9eed_5b3539718a70.slice/crio-1db2604437c2bd3b13c9bc437b3f67904997ac66c969bf42c313b48bf8ce894a WatchSource:0}: Error finding container 1db2604437c2bd3b13c9bc437b3f67904997ac66c969bf42c313b48bf8ce894a: Status 404 returned error can't find the container with id 1db2604437c2bd3b13c9bc437b3f67904997ac66c969bf42c313b48bf8ce894a
Sep 15 21:10:13.440255 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:13.440228    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dc6096_8902_4327_b266_0d73cb2f676d.slice/crio-9644792e07dcf9e950bf711b575f14efffe084caefbeb05c4b09a283c400411b WatchSource:0}: Error finding container 9644792e07dcf9e950bf711b575f14efffe084caefbeb05c4b09a283c400411b: Status 404 returned error can't find the container with id 9644792e07dcf9e950bf711b575f14efffe084caefbeb05c4b09a283c400411b
Sep 15 21:10:13.440876 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:13.440855    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b009b6_fdf0_47c4_bff1_de5e0c1db03e.slice/crio-8050ecb1aad1b2603fdec9539f3bcdfda6fbeb0265fdcac8039c3201c887bb79 WatchSource:0}: Error finding container 8050ecb1aad1b2603fdec9539f3bcdfda6fbeb0265fdcac8039c3201c887bb79: Status 404 returned error can't find the container with id 8050ecb1aad1b2603fdec9539f3bcdfda6fbeb0265fdcac8039c3201c887bb79
Sep 15 21:10:13.441729 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:13.441643    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd798c0_c26c_4ac1_9ae0_e5023908313b.slice/crio-437a520fd4763ac9ad0acabf5d19cb13e6c93b7456831e96553cf58424981112 WatchSource:0}: Error finding container 437a520fd4763ac9ad0acabf5d19cb13e6c93b7456831e96553cf58424981112: Status 404 returned error can't find the container with id 437a520fd4763ac9ad0acabf5d19cb13e6c93b7456831e96553cf58424981112
Sep 15 21:10:13.442464 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:13.442394    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc240ac_5432_43b5_ba3d_843e08df0eb9.slice/crio-5dcab69c17ffc6f1e0336b9a6b920d10753488b8dff6747c68ca404e76c3bc76 WatchSource:0}: Error finding container 5dcab69c17ffc6f1e0336b9a6b920d10753488b8dff6747c68ca404e76c3bc76: Status 404 returned error can't find the container with id 5dcab69c17ffc6f1e0336b9a6b920d10753488b8dff6747c68ca404e76c3bc76
Sep 15 21:10:13.443459 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:13.443440    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8060115d_0d03_495c_bd01_c0a4315c501f.slice/crio-8d8f6c4894c534e73dd94312d7e2dde3eef73d31786f27bc0aa9854ec02f4870 WatchSource:0}: Error finding container 8d8f6c4894c534e73dd94312d7e2dde3eef73d31786f27bc0aa9854ec02f4870: Status 404 returned error can't find the container with id 8d8f6c4894c534e73dd94312d7e2dde3eef73d31786f27bc0aa9854ec02f4870
Sep 15 21:10:13.462880 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.462742    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:05:12 +0000 UTC" deadline="2028-07-03 05:43:09.477308144 +0000 UTC"
Sep 15 21:10:13.462940 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.462881    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15752h32m56.014431578s"
Sep 15 21:10:13.520090 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.520071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:13.520207 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:13.520186    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:13.527827 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.527805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ltn26" event={"ID":"2bd798c0-c26c-4ac1-9ae0-e5023908313b","Type":"ContainerStarted","Data":"437a520fd4763ac9ad0acabf5d19cb13e6c93b7456831e96553cf58424981112"}
Sep 15 21:10:13.528933 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.528910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck" event={"ID":"18b009b6-fdf0-47c4-bff1-de5e0c1db03e","Type":"ContainerStarted","Data":"8050ecb1aad1b2603fdec9539f3bcdfda6fbeb0265fdcac8039c3201c887bb79"}
Sep 15 21:10:13.529984 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.529966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hcnhm" event={"ID":"61e8e166-9aa3-4f53-b9d2-564ce990235c","Type":"ContainerStarted","Data":"1a3d2658520eae2a3c4cdfd89823183fbd908658dfe492371c39ab8f8deec220"}
Sep 15 21:10:13.531214 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.531190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wqnjj" event={"ID":"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6","Type":"ContainerStarted","Data":"166ac5b04f612b37e73f94c5f78239a3155e70948a0b861ba19de47a29f34e67"}
Sep 15 21:10:13.532417 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.532375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sds6c" event={"ID":"8060115d-0d03-495c-bd01-c0a4315c501f","Type":"ContainerStarted","Data":"8d8f6c4894c534e73dd94312d7e2dde3eef73d31786f27bc0aa9854ec02f4870"}
Sep 15 21:10:13.533567 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.533540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn" event={"ID":"cbc240ac-5432-43b5-ba3d-843e08df0eb9","Type":"ContainerStarted","Data":"5dcab69c17ffc6f1e0336b9a6b920d10753488b8dff6747c68ca404e76c3bc76"}
Sep 15 21:10:13.536581 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.536557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc8jl" event={"ID":"51dc6096-8902-4327-b266-0d73cb2f676d","Type":"ContainerStarted","Data":"9644792e07dcf9e950bf711b575f14efffe084caefbeb05c4b09a283c400411b"}
Sep 15 21:10:13.538045 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.538024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b5ndc" event={"ID":"d76ae1c9-7503-48a0-9eed-5b3539718a70","Type":"ContainerStarted","Data":"1db2604437c2bd3b13c9bc437b3f67904997ac66c969bf42c313b48bf8ce894a"}
Sep 15 21:10:13.539244 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.539220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kdr47" event={"ID":"30f82454-0b7e-4f7b-b186-b44771628f4b","Type":"ContainerStarted","Data":"8c58ac2f04e75ccad8ef4f8d509d1d2641f35764906e9fca785493db93f7d1e0"}
Sep 15 21:10:13.541208 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.541183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal" event={"ID":"bfa4cb4183b054641aa8c971a669467e","Type":"ContainerStarted","Data":"2bce9d3110441aee9251877d51a8e3f8beeed4c934244dd08d13265f24d9ee0a"}
Sep 15 21:10:13.557928 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:13.557877    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-191.ec2.internal" podStartSLOduration=2.557859781 podStartE2EDuration="2.557859781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:13.55428888 +0000 UTC m=+3.517346261" watchObservedRunningTime="2026-09-15 21:10:13.557859781 +0000 UTC m=+3.520917162"
Sep 15 21:10:14.053098 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:14.053020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:14.053251 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:14.053197    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:14.053319 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:14.053261    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret podName:e74e7fd1-2672-4c85-b798-a3f036fb7681 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:16.053242792 +0000 UTC m=+6.016300168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret") pod "global-pull-secret-syncer-2mthm" (UID: "e74e7fd1-2672-4c85-b798-a3f036fb7681") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:14.153711 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:14.153673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:14.153876 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:14.153741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdd9\" (UniqueName: \"kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9\") pod \"network-check-target-z477j\" (UID: \"4832498c-7ee0-4464-aae8-919202c79bdd\") " pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:14.153941 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:14.153875    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:14.153941 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:14.153892    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:14.153941 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:14.153904    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfdd9 for pod openshift-network-diagnostics/network-check-target-z477j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:14.154067 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:14.153959    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9 podName:4832498c-7ee0-4464-aae8-919202c79bdd nodeName:}" failed. No retries permitted until 2026-09-15 21:10:16.153941609 +0000 UTC m=+6.116998989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfdd9" (UniqueName: "kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9") pod "network-check-target-z477j" (UID: "4832498c-7ee0-4464-aae8-919202c79bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:14.154365 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:14.154329    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:14.154474 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:14.154396    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs podName:a25ac072-b1cb-4739-a1e7-f0ed49ddf881 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:16.154382267 +0000 UTC m=+6.117439643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs") pod "network-metrics-daemon-gp2b5" (UID: "a25ac072-b1cb-4739-a1e7-f0ed49ddf881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:14.463211 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:14.463039    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:05:12 +0000 UTC" deadline="2028-06-26 06:54:44.125687332 +0000 UTC"
Sep 15 21:10:14.463211 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:14.463096    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15585h44m29.662595851s"
Sep 15 21:10:14.522703 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:14.522654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:14.522840 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:14.522823    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:14.522968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:14.522953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:14.523074 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:14.523045    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:14.558813 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:14.557778    2563 generic.go:356] "Generic (PLEG): container finished" podID="785230b8847b3f5457cd788cddf1108f" containerID="7a3b97bd163410416f7be07a5226a2438036cc3ee4be049dc4242f417ad69cbd" exitCode=0
Sep 15 21:10:14.558813 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:14.558630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal" event={"ID":"785230b8847b3f5457cd788cddf1108f","Type":"ContainerDied","Data":"7a3b97bd163410416f7be07a5226a2438036cc3ee4be049dc4242f417ad69cbd"}
Sep 15 21:10:15.520488 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:15.519982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:15.520488 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:15.520100    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:15.575032 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:15.574989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal" event={"ID":"785230b8847b3f5457cd788cddf1108f","Type":"ContainerStarted","Data":"272ca773e2396f72f3786b2edd2ab89a352a7bb42257fb038f67c8cfa8ae0d0b"}
Sep 15 21:10:16.068199 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:16.067641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:16.068199 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:16.067813    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:16.068199 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:16.067871    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret podName:e74e7fd1-2672-4c85-b798-a3f036fb7681 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:20.067854256 +0000 UTC m=+10.030911618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret") pod "global-pull-secret-syncer-2mthm" (UID: "e74e7fd1-2672-4c85-b798-a3f036fb7681") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:16.168725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:16.168564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:16.168725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:16.168640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdd9\" (UniqueName: \"kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9\") pod \"network-check-target-z477j\" (UID: \"4832498c-7ee0-4464-aae8-919202c79bdd\") " pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:16.168938 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:16.168786    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:16.168938 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:16.168848    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs podName:a25ac072-b1cb-4739-a1e7-f0ed49ddf881 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:20.168828669 +0000 UTC m=+10.131886044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs") pod "network-metrics-daemon-gp2b5" (UID: "a25ac072-b1cb-4739-a1e7-f0ed49ddf881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:16.169634 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:16.169260    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:16.169634 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:16.169284    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:16.169634 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:16.169296    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfdd9 for pod openshift-network-diagnostics/network-check-target-z477j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:16.169634 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:16.169370    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9 podName:4832498c-7ee0-4464-aae8-919202c79bdd nodeName:}" failed. No retries permitted until 2026-09-15 21:10:20.169336658 +0000 UTC m=+10.132394035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfdd9" (UniqueName: "kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9") pod "network-check-target-z477j" (UID: "4832498c-7ee0-4464-aae8-919202c79bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:16.521803 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:16.521250    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:16.521803 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:16.521394    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:16.522315 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:16.521814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:16.522315 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:16.521929    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:17.520278 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:17.520219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:17.520460 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:17.520391    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:18.520549 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:18.520241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:18.520549 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:18.520390    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:18.521030 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:18.520737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:18.521030 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:18.520833    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:19.519999 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:19.519920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:19.520139 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:19.520053    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:20.101435 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:20.101360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:20.101935 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:20.101483    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:20.101935 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:20.101552    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret podName:e74e7fd1-2672-4c85-b798-a3f036fb7681 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:28.101532867 +0000 UTC m=+18.064590242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret") pod "global-pull-secret-syncer-2mthm" (UID: "e74e7fd1-2672-4c85-b798-a3f036fb7681") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:20.202287 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:20.202204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:20.202515 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:20.202318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdd9\" (UniqueName: \"kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9\") pod \"network-check-target-z477j\" (UID: \"4832498c-7ee0-4464-aae8-919202c79bdd\") " pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:20.202515 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:20.202487    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:20.202515 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:20.202505    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:20.202515 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:20.202516    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfdd9 for pod openshift-network-diagnostics/network-check-target-z477j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:20.202774 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:20.202569    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9 podName:4832498c-7ee0-4464-aae8-919202c79bdd nodeName:}" failed. No retries permitted until 2026-09-15 21:10:28.202550897 +0000 UTC m=+18.165608277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfdd9" (UniqueName: "kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9") pod "network-check-target-z477j" (UID: "4832498c-7ee0-4464-aae8-919202c79bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:20.203042 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:20.202956    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:20.203042 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:20.203022    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs podName:a25ac072-b1cb-4739-a1e7-f0ed49ddf881 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:28.203006268 +0000 UTC m=+18.166063631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs") pod "network-metrics-daemon-gp2b5" (UID: "a25ac072-b1cb-4739-a1e7-f0ed49ddf881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:20.521480 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:20.521405    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:20.521625 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:20.521509    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:20.521812 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:20.521790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:20.521911 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:20.521892    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:21.520028 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:21.519993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:21.520505 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:21.520121    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:22.519376 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:22.519331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:22.519562 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:22.519471    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:22.519562 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:22.519537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:22.519706 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:22.519679    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:23.519527 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:23.519504    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:23.519826 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:23.519592    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:24.519551 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:24.519501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:24.519551 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:24.519528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:24.520018 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:24.519634    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:24.520018 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:24.519743    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:25.519707 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:25.519672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:25.520107 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:25.519774    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:26.520206 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:26.520168    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:26.520641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:26.520216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:26.520641 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:26.520312    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:26.520641 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:26.520442    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:27.519774 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:27.519741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:27.519986 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:27.519862    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:28.162289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:28.162256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:28.162683 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:28.162386    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:28.162683 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:28.162449    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret podName:e74e7fd1-2672-4c85-b798-a3f036fb7681 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:44.162431666 +0000 UTC m=+34.125489047 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret") pod "global-pull-secret-syncer-2mthm" (UID: "e74e7fd1-2672-4c85-b798-a3f036fb7681") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:28.262834 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:28.262802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdd9\" (UniqueName: \"kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9\") pod \"network-check-target-z477j\" (UID: \"4832498c-7ee0-4464-aae8-919202c79bdd\") " pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:28.263002 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:28.262862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:28.263002 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:28.262958    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:28.263002 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:28.262976    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:28.263002 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:28.262999    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:28.263193 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:28.263009    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfdd9 for pod openshift-network-diagnostics/network-check-target-z477j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:28.263193 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:28.263019    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs podName:a25ac072-b1cb-4739-a1e7-f0ed49ddf881 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:44.26300035 +0000 UTC m=+34.226057710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs") pod "network-metrics-daemon-gp2b5" (UID: "a25ac072-b1cb-4739-a1e7-f0ed49ddf881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:28.263193 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:28.263060    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9 podName:4832498c-7ee0-4464-aae8-919202c79bdd nodeName:}" failed. No retries permitted until 2026-09-15 21:10:44.263043425 +0000 UTC m=+34.226100807 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfdd9" (UniqueName: "kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9") pod "network-check-target-z477j" (UID: "4832498c-7ee0-4464-aae8-919202c79bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:28.520177 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:28.520102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:28.520319 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:28.520106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:28.520319 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:28.520299    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:28.520461 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:28.520425    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:29.519650 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:29.519456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:29.520021 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:29.519750    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:30.520835 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:30.520805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:30.521206 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:30.520908    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:30.521206 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:30.520964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:30.521206 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:30.521078    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:31.519848 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.519641    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:31.519999 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:31.519939    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:31.608189 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.608112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sds6c" event={"ID":"8060115d-0d03-495c-bd01-c0a4315c501f","Type":"ContainerStarted","Data":"fdf4131e3dc0126b6b54f9a14b1016272a45a1f55f98862a6b042c76e00ceda9"}
Sep 15 21:10:31.611607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.611580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn" event={"ID":"cbc240ac-5432-43b5-ba3d-843e08df0eb9","Type":"ContainerStarted","Data":"984b7d5426f2eb38238f1c117b85108d38e9738ecfd13caebd6a3959ba795958"}
Sep 15 21:10:31.611737 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.611661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn" event={"ID":"cbc240ac-5432-43b5-ba3d-843e08df0eb9","Type":"ContainerStarted","Data":"1e9390631221e6cf62dabae834babd37a30eb70a319b37d6c31c15e1cfd1f21f"}
Sep 15 21:10:31.611737 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.611713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn" event={"ID":"cbc240ac-5432-43b5-ba3d-843e08df0eb9","Type":"ContainerStarted","Data":"d5f18168082770034f8ad42402ec65c8812f004e8451411a34fd161e296ff5b4"}
Sep 15 21:10:31.611919 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.611770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn" event={"ID":"cbc240ac-5432-43b5-ba3d-843e08df0eb9","Type":"ContainerStarted","Data":"f06ae7d961f24e9167853d869d27dbee8ace49c87227037570ceb847910c6d5b"}
Sep 15 21:10:31.611919 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.611825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn" event={"ID":"cbc240ac-5432-43b5-ba3d-843e08df0eb9","Type":"ContainerStarted","Data":"dd587bd199b4dfb4a3f52bd7e36fa78f9d9c075340d3283273424f1bac5258ee"}
Sep 15 21:10:31.611919 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.611856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn" event={"ID":"cbc240ac-5432-43b5-ba3d-843e08df0eb9","Type":"ContainerStarted","Data":"0387fe31c014bd78dcce2f47f28d2ba10852e98b3a78c6e3031b9ba9140af12f"}
Sep 15 21:10:31.613305 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.613280    2563 generic.go:356] "Generic (PLEG): container finished" podID="51dc6096-8902-4327-b266-0d73cb2f676d" containerID="19d024c21c3b83fd5256f25211e60c1c94e7b5ec45d3e30413da145364e1a868" exitCode=0
Sep 15 21:10:31.613433 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.613334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc8jl" event={"ID":"51dc6096-8902-4327-b266-0d73cb2f676d","Type":"ContainerDied","Data":"19d024c21c3b83fd5256f25211e60c1c94e7b5ec45d3e30413da145364e1a868"}
Sep 15 21:10:31.614911 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.614892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b5ndc" event={"ID":"d76ae1c9-7503-48a0-9eed-5b3539718a70","Type":"ContainerStarted","Data":"775c2168180f3a8cbab1aa6913a0f309a5ae5e579ab854c2b7d34af6afdd559b"}
Sep 15 21:10:31.616663 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.616642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ltn26" event={"ID":"2bd798c0-c26c-4ac1-9ae0-e5023908313b","Type":"ContainerStarted","Data":"69c503170c410d4213b20fc0f4b466fbbbff8a03003f053f9c9bccc4ff5ca1a8"}
Sep 15 21:10:31.618279 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.618254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck" event={"ID":"18b009b6-fdf0-47c4-bff1-de5e0c1db03e","Type":"ContainerStarted","Data":"edc6f8e0433265c1f58451b11c5a2791de723a84535e12f9f0c14728901fbc90"}
Sep 15 21:10:31.619855 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.619831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hcnhm" event={"ID":"61e8e166-9aa3-4f53-b9d2-564ce990235c","Type":"ContainerStarted","Data":"d4ec800c1cce650a7ee477204fd0289a4948d379dd612ac4577760421ee376c3"}
Sep 15 21:10:31.621466 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.621446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wqnjj" event={"ID":"e6a45cb7-2e0d-4e7a-b855-3d40b108cdc6","Type":"ContainerStarted","Data":"35fd78848e01ec70c884fe03809fa72dd07e3e9ac0c9172e20623b8b6f48a1d7"}
Sep 15 21:10:31.624483 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.624447    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sds6c" podStartSLOduration=4.358087826 podStartE2EDuration="21.624432954s" totalImagesPullingTime="17.266345128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:31.623690638 +0000 UTC m=+21.586748018" watchObservedRunningTime="2026-09-15 21:10:31.624432954 +0000 UTC m=+21.587490336"
Sep 15 21:10:31.624592 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.624568    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-191.ec2.internal" podStartSLOduration=20.624560312 podStartE2EDuration="20.624560312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:15.589218188 +0000 UTC m=+5.552275568" watchObservedRunningTime="2026-09-15 21:10:31.624560312 +0000 UTC m=+21.587617694"
Sep 15 21:10:31.640124 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.640081    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wqnjj" podStartSLOduration=4.281950407 podStartE2EDuration="21.640069663s" totalImagesPullingTime="17.358119256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:31.63980869 +0000 UTC m=+21.602866071" watchObservedRunningTime="2026-09-15 21:10:31.640069663 +0000 UTC m=+21.603127046"
Sep 15 21:10:31.741833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.741788    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b5ndc" podStartSLOduration=4.472639232 podStartE2EDuration="21.74177286s" totalImagesPullingTime="17.269133628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:31.741755625 +0000 UTC m=+21.704813006" watchObservedRunningTime="2026-09-15 21:10:31.74177286 +0000 UTC m=+21.704830240"
Sep 15 21:10:31.741986 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.741905    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ltn26" podStartSLOduration=4.462385524 podStartE2EDuration="21.741900559s" totalImagesPullingTime="17.279515035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:31.720117066 +0000 UTC m=+21.683174449" watchObservedRunningTime="2026-09-15 21:10:31.741900559 +0000 UTC m=+21.704957939"
Sep 15 21:10:31.758694 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:31.758654    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hcnhm" podStartSLOduration=4.469636482 podStartE2EDuration="21.758639571s" totalImagesPullingTime="17.289003089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:31.758211958 +0000 UTC m=+21.721269337" watchObservedRunningTime="2026-09-15 21:10:31.758639571 +0000 UTC m=+21.721696952"
Sep 15 21:10:32.132735 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:32.132661    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 21:10:32.499395 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:32.499234    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T21:10:32.132675033Z","UUID":"78dc7b1d-7cc9-4738-b422-ad415a6fa222","Handler":null,"Name":"","Endpoint":""}
Sep 15 21:10:32.501059 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:32.500885    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 21:10:32.501177 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:32.501092    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 21:10:32.519745 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:32.519720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:32.519745 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:32.519736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:32.519910 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:32.519842    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:32.520006 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:32.519988    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:32.625904 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:32.625875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck" event={"ID":"18b009b6-fdf0-47c4-bff1-de5e0c1db03e","Type":"ContainerStarted","Data":"ac38b688c41af97920b1f0ba65f25d7446383c619486f163faeddc3851ce239c"}
Sep 15 21:10:32.627631 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:32.627593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kdr47" event={"ID":"30f82454-0b7e-4f7b-b186-b44771628f4b","Type":"ContainerStarted","Data":"a9d5bfd6e77705cc0f46faa0c7ae09f1a676abe3e275ec2739ef8acb5e6dba6c"}
Sep 15 21:10:32.643956 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:32.643917    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kdr47" podStartSLOduration=5.353910641 podStartE2EDuration="22.643903105s" totalImagesPullingTime="17.289992464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:32.643759018 +0000 UTC m=+22.606816399" watchObservedRunningTime="2026-09-15 21:10:32.643903105 +0000 UTC m=+22.606960486"
Sep 15 21:10:33.519686 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:33.519658    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:33.519832 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:33.519780    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:33.632932 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:33.632853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck" event={"ID":"18b009b6-fdf0-47c4-bff1-de5e0c1db03e","Type":"ContainerStarted","Data":"4748c345aa0b60d6530d70f257681ec5d69ce07e8c479535b86088ccdd15781f"}
Sep 15 21:10:33.636586 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:33.636558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn" event={"ID":"cbc240ac-5432-43b5-ba3d-843e08df0eb9","Type":"ContainerStarted","Data":"7b4e27b50b7f68c55cb3f8a4ee879a4ac7f8b2bc7b56b7a71a530ca717e089bb"}
Sep 15 21:10:33.651161 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:33.651118    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blfck" podStartSLOduration=4.0843696959999995 podStartE2EDuration="23.651104448s" totalImagesPullingTime="19.566734752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:33.650838712 +0000 UTC m=+23.613896095" watchObservedRunningTime="2026-09-15 21:10:33.651104448 +0000 UTC m=+23.614161880"
Sep 15 21:10:34.519674 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:34.519631    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:34.519939 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:34.519761    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:34.519939 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:34.519802    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:34.519939 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:34.519869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:35.519977 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:35.519941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:35.520440 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:35.520076    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:35.769413 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:35.769388    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:35.769924 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:35.769904    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:36.520219 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:36.519981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:36.520722 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:36.520041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:36.520722 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:36.520257    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:36.520722 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:36.520316    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:36.644983 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:36.644957    2563 generic.go:356] "Generic (PLEG): container finished" podID="51dc6096-8902-4327-b266-0d73cb2f676d" containerID="e02074511e9328de5b3a19ced1d8eba92849bf194694ab4a2665cee8fe0583c0" exitCode=0
Sep 15 21:10:36.645128 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:36.645044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc8jl" event={"ID":"51dc6096-8902-4327-b266-0d73cb2f676d","Type":"ContainerDied","Data":"e02074511e9328de5b3a19ced1d8eba92849bf194694ab4a2665cee8fe0583c0"}
Sep 15 21:10:36.648843 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:36.648811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn" event={"ID":"cbc240ac-5432-43b5-ba3d-843e08df0eb9","Type":"ContainerStarted","Data":"15c81cfab386ca23aa5e3b9749740e05be4c65c575d3a9a4b28519658a2ca580"}
Sep 15 21:10:36.648948 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:36.648924    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:36.649442 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:36.649424    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b5ndc"
Sep 15 21:10:36.701097 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:36.701050    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn" podStartSLOduration=9.377215471 podStartE2EDuration="26.701036788s" totalImagesPullingTime="17.323821317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:36.699890479 +0000 UTC m=+26.662947900" watchObservedRunningTime="2026-09-15 21:10:36.701036788 +0000 UTC m=+26.664094168"
Sep 15 21:10:37.519998 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.519973    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:37.520132 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:37.520072    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:37.654729 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.654703    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:37.655485 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.654791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:37.655485 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.654842    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:37.657074 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.657047    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2mthm"]
Sep 15 21:10:37.657157 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.657142    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:37.657257 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:37.657238    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:37.660121 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.660079    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z477j"]
Sep 15 21:10:37.660204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.660167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:37.660263 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:37.660231    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:37.671157 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.671138    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:37.671261 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.671252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:10:37.671324 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.671302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gp2b5"]
Sep 15 21:10:37.671401 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:37.671390    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:37.671513 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:37.671491    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:38.658147 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:38.658114    2563 generic.go:356] "Generic (PLEG): container finished" podID="51dc6096-8902-4327-b266-0d73cb2f676d" containerID="c5e31639c1a931b9418e931a0989c857397403bca27c6ac6cb9c0f7684651993" exitCode=0
Sep 15 21:10:38.658603 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:38.658196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc8jl" event={"ID":"51dc6096-8902-4327-b266-0d73cb2f676d","Type":"ContainerDied","Data":"c5e31639c1a931b9418e931a0989c857397403bca27c6ac6cb9c0f7684651993"}
Sep 15 21:10:39.519738 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:39.519553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:39.519894 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:39.519595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:39.519894 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:39.519817    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:39.519894 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:39.519595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:39.520000 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:39.519886    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:39.520000 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:39.519955    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:40.665587 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:40.665555    2563 generic.go:356] "Generic (PLEG): container finished" podID="51dc6096-8902-4327-b266-0d73cb2f676d" containerID="686e073dd5d9973a585df7d15d745591e5b22617019c4a8bf2a1406e0c23b8db" exitCode=0
Sep 15 21:10:40.665934 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:40.665594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc8jl" event={"ID":"51dc6096-8902-4327-b266-0d73cb2f676d","Type":"ContainerDied","Data":"686e073dd5d9973a585df7d15d745591e5b22617019c4a8bf2a1406e0c23b8db"}
Sep 15 21:10:41.519620 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:41.519580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:41.519831 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:41.519626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:41.519831 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:41.519682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:41.519831 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:41.519702    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:41.519831 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:41.519770    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:41.519966 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:41.519851    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:43.519986 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:43.519954    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:43.520487 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:43.519995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:43.520487 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:43.520045    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z477j" podUID="4832498c-7ee0-4464-aae8-919202c79bdd"
Sep 15 21:10:43.520487 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:43.520109    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gp2b5" podUID="a25ac072-b1cb-4739-a1e7-f0ed49ddf881"
Sep 15 21:10:43.520487 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:43.520155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:43.520487 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:43.520244    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2mthm" podUID="e74e7fd1-2672-4c85-b798-a3f036fb7681"
Sep 15 21:10:44.183460 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.183431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:44.183643 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.183537    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:44.183643 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.183590    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret podName:e74e7fd1-2672-4c85-b798-a3f036fb7681 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:16.183573968 +0000 UTC m=+66.146631345 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret") pod "global-pull-secret-syncer-2mthm" (UID: "e74e7fd1-2672-4c85-b798-a3f036fb7681") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:44.284123 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.284093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:44.284245 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.284164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdd9\" (UniqueName: \"kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9\") pod \"network-check-target-z477j\" (UID: \"4832498c-7ee0-4464-aae8-919202c79bdd\") " pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:44.284303 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.284250    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:44.284303 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.284273    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:44.284303 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.284290    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:44.284303 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.284303    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfdd9 for pod openshift-network-diagnostics/network-check-target-z477j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:44.284509 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.284323    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs podName:a25ac072-b1cb-4739-a1e7-f0ed49ddf881 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:16.284302993 +0000 UTC m=+66.247360372 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs") pod "network-metrics-daemon-gp2b5" (UID: "a25ac072-b1cb-4739-a1e7-f0ed49ddf881") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:44.284509 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.284369    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9 podName:4832498c-7ee0-4464-aae8-919202c79bdd nodeName:}" failed. No retries permitted until 2026-09-15 21:11:16.284336799 +0000 UTC m=+66.247394175 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfdd9" (UniqueName: "kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9") pod "network-check-target-z477j" (UID: "4832498c-7ee0-4464-aae8-919202c79bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:44.294886 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.294868    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-191.ec2.internal" event="NodeReady"
Sep 15 21:10:44.295035 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.295024    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 21:10:44.329764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.329726    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6848588594-gj864"]
Sep 15 21:10:44.332300 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.332283    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.334274 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.334254    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 21:10:44.334403 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.334325    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dnxk5\""
Sep 15 21:10:44.334403 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.334341    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 21:10:44.334403 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.334366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 21:10:44.343481 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.343333    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2gb66"]
Sep 15 21:10:44.344136 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.344114    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 21:10:44.345856 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.345839    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m9jvn"]
Sep 15 21:10:44.345985 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.345960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2gb66"
Sep 15 21:10:44.347717 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.347701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.347931 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.347739    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6848588594-gj864"]
Sep 15 21:10:44.348580 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.348560    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 21:10:44.348675 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.348623    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 21:10:44.348841 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.348811    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8lhn5\""
Sep 15 21:10:44.348908 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.348868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 21:10:44.349796 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.349566    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 21:10:44.349888 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.349839    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 21:10:44.350378 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.350315    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nfdpp\""
Sep 15 21:10:44.351035 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.350994    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2gb66"]
Sep 15 21:10:44.351879 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.351860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m9jvn"]
Sep 15 21:10:44.485588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-certificates\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.485588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-trusted-ca\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.485588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/04d67dbc-688d-42da-a42a-66aa0b7258e8-config-volume\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.485814 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.485814 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d67dbc-688d-42da-a42a-66aa0b7258e8-tmp-dir\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.485814 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-image-registry-private-configuration\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.485814 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdgh\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-kube-api-access-4jdgh\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.485814 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/691cfa1a-d482-4ccb-91d4-e0dc621dc229-cert\") pod \"ingress-canary-2gb66\" (UID: \"691cfa1a-d482-4ccb-91d4-e0dc621dc229\") " pod="openshift-ingress-canary/ingress-canary-2gb66"
Sep 15 21:10:44.485992 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb79s\" (UniqueName: \"kubernetes.io/projected/691cfa1a-d482-4ccb-91d4-e0dc621dc229-kube-api-access-fb79s\") pod \"ingress-canary-2gb66\" (UID: \"691cfa1a-d482-4ccb-91d4-e0dc621dc229\") " pod="openshift-ingress-canary/ingress-canary-2gb66"
Sep 15 21:10:44.485992 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-installation-pull-secrets\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.485992 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.485992 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zq6\" (UniqueName: \"kubernetes.io/projected/04d67dbc-688d-42da-a42a-66aa0b7258e8-kube-api-access-86zq6\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.485992 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7835480-8a67-45e3-a4bf-aafaba2af2bb-ca-trust-extracted\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.485992 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.485979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-bound-sa-token\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.587234 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-image-registry-private-configuration\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.587668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdgh\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-kube-api-access-4jdgh\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.587668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/691cfa1a-d482-4ccb-91d4-e0dc621dc229-cert\") pod \"ingress-canary-2gb66\" (UID: \"691cfa1a-d482-4ccb-91d4-e0dc621dc229\") " pod="openshift-ingress-canary/ingress-canary-2gb66"
Sep 15 21:10:44.587668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb79s\" (UniqueName: \"kubernetes.io/projected/691cfa1a-d482-4ccb-91d4-e0dc621dc229-kube-api-access-fb79s\") pod \"ingress-canary-2gb66\" (UID: \"691cfa1a-d482-4ccb-91d4-e0dc621dc229\") " pod="openshift-ingress-canary/ingress-canary-2gb66"
Sep 15 21:10:44.587668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-installation-pull-secrets\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.587668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.587668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zq6\" (UniqueName: \"kubernetes.io/projected/04d67dbc-688d-42da-a42a-66aa0b7258e8-kube-api-access-86zq6\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.587668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7835480-8a67-45e3-a4bf-aafaba2af2bb-ca-trust-extracted\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.587668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-bound-sa-token\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.587668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-certificates\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.587668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-trusted-ca\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.588154 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.587685    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:10:44.588154 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.587699    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6848588594-gj864: secret "image-registry-tls" not found
Sep 15 21:10:44.588154 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/04d67dbc-688d-42da-a42a-66aa0b7258e8-config-volume\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.588154 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.588154 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d67dbc-688d-42da-a42a-66aa0b7258e8-tmp-dir\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.588154 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.587760    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls podName:f7835480-8a67-45e3-a4bf-aafaba2af2bb nodeName:}" failed. No retries permitted until 2026-09-15 21:10:45.087741433 +0000 UTC m=+35.050798792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls") pod "image-registry-6848588594-gj864" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb") : secret "image-registry-tls" not found
Sep 15 21:10:44.588154 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7835480-8a67-45e3-a4bf-aafaba2af2bb-ca-trust-extracted\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.588154 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.587991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d67dbc-688d-42da-a42a-66aa0b7258e8-tmp-dir\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.588154 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.588090    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:10:44.588154 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:44.588131    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls podName:04d67dbc-688d-42da-a42a-66aa0b7258e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:45.088118081 +0000 UTC m=+35.051175444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls") pod "dns-default-m9jvn" (UID: "04d67dbc-688d-42da-a42a-66aa0b7258e8") : secret "dns-default-metrics-tls" not found
Sep 15 21:10:44.588604 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.588476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-certificates\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.588604 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.588493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/04d67dbc-688d-42da-a42a-66aa0b7258e8-config-volume\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.589033 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.589011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-trusted-ca\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.591698 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.591676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-image-registry-private-configuration\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.591698 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.591687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-installation-pull-secrets\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.591895 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.591813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/691cfa1a-d482-4ccb-91d4-e0dc621dc229-cert\") pod \"ingress-canary-2gb66\" (UID: \"691cfa1a-d482-4ccb-91d4-e0dc621dc229\") " pod="openshift-ingress-canary/ingress-canary-2gb66"
Sep 15 21:10:44.596617 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.596591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zq6\" (UniqueName: \"kubernetes.io/projected/04d67dbc-688d-42da-a42a-66aa0b7258e8-kube-api-access-86zq6\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:44.596725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.596595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb79s\" (UniqueName: \"kubernetes.io/projected/691cfa1a-d482-4ccb-91d4-e0dc621dc229-kube-api-access-fb79s\") pod \"ingress-canary-2gb66\" (UID: \"691cfa1a-d482-4ccb-91d4-e0dc621dc229\") " pod="openshift-ingress-canary/ingress-canary-2gb66"
Sep 15 21:10:44.596725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.596676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-bound-sa-token\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.596902 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.596886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdgh\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-kube-api-access-4jdgh\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:44.658677 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:44.658654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2gb66"
Sep 15 21:10:45.092284 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.092249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:45.092484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.092326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:45.092484 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:45.092434    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:10:45.092484 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:45.092460    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6848588594-gj864: secret "image-registry-tls" not found
Sep 15 21:10:45.092484 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:45.092434    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:10:45.092649 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:45.092531    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls podName:f7835480-8a67-45e3-a4bf-aafaba2af2bb nodeName:}" failed. No retries permitted until 2026-09-15 21:10:46.092510352 +0000 UTC m=+36.055567716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls") pod "image-registry-6848588594-gj864" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb") : secret "image-registry-tls" not found
Sep 15 21:10:45.092649 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:45.092550    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls podName:04d67dbc-688d-42da-a42a-66aa0b7258e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:46.092539601 +0000 UTC m=+36.055596965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls") pod "dns-default-m9jvn" (UID: "04d67dbc-688d-42da-a42a-66aa0b7258e8") : secret "dns-default-metrics-tls" not found
Sep 15 21:10:45.428172 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.428023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2gb66"]
Sep 15 21:10:45.446057 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:10:45.446032    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691cfa1a_d482_4ccb_91d4_e0dc621dc229.slice/crio-14de1cea97960ff8fa867ccdd5cf027c5a3bc2e69ce78485e62dde9d79b1fcc0 WatchSource:0}: Error finding container 14de1cea97960ff8fa867ccdd5cf027c5a3bc2e69ce78485e62dde9d79b1fcc0: Status 404 returned error can't find the container with id 14de1cea97960ff8fa867ccdd5cf027c5a3bc2e69ce78485e62dde9d79b1fcc0
Sep 15 21:10:45.520200 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.520174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:10:45.520278 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.520202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:10:45.520278 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.520174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:10:45.522758 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.522727    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:10:45.522758 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.522752    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:10:45.522912 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.522779    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:10:45.523341 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.523325    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6cwlg\""
Sep 15 21:10:45.523524 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.523486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:10:45.523602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.523552    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8fnh6\""
Sep 15 21:10:45.682437 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.682409    2563 generic.go:356] "Generic (PLEG): container finished" podID="51dc6096-8902-4327-b266-0d73cb2f676d" containerID="13572d0d980610e4c4affedfb0adef93f51ff4984fb2fc93ab45d8bfbf2b5acc" exitCode=0
Sep 15 21:10:45.682975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.682492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc8jl" event={"ID":"51dc6096-8902-4327-b266-0d73cb2f676d","Type":"ContainerDied","Data":"13572d0d980610e4c4affedfb0adef93f51ff4984fb2fc93ab45d8bfbf2b5acc"}
Sep 15 21:10:45.683683 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:45.683654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2gb66" event={"ID":"691cfa1a-d482-4ccb-91d4-e0dc621dc229","Type":"ContainerStarted","Data":"14de1cea97960ff8fa867ccdd5cf027c5a3bc2e69ce78485e62dde9d79b1fcc0"}
Sep 15 21:10:46.099714 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:46.099633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:46.099714 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:46.099693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:46.099915 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:46.099806    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:10:46.099915 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:46.099831    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:10:46.099915 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:46.099842    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6848588594-gj864: secret "image-registry-tls" not found
Sep 15 21:10:46.099915 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:46.099871    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls podName:04d67dbc-688d-42da-a42a-66aa0b7258e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:48.099852233 +0000 UTC m=+38.062909610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls") pod "dns-default-m9jvn" (UID: "04d67dbc-688d-42da-a42a-66aa0b7258e8") : secret "dns-default-metrics-tls" not found
Sep 15 21:10:46.099915 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:46.099889    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls podName:f7835480-8a67-45e3-a4bf-aafaba2af2bb nodeName:}" failed. No retries permitted until 2026-09-15 21:10:48.099880202 +0000 UTC m=+38.062937566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls") pod "image-registry-6848588594-gj864" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb") : secret "image-registry-tls" not found
Sep 15 21:10:46.690276 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:46.690241    2563 generic.go:356] "Generic (PLEG): container finished" podID="51dc6096-8902-4327-b266-0d73cb2f676d" containerID="3a4f72038b442d065184051ee3e9c0d36022f0c5f9cbfbffda1d05b06311c014" exitCode=0
Sep 15 21:10:46.690788 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:46.690292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc8jl" event={"ID":"51dc6096-8902-4327-b266-0d73cb2f676d","Type":"ContainerDied","Data":"3a4f72038b442d065184051ee3e9c0d36022f0c5f9cbfbffda1d05b06311c014"}
Sep 15 21:10:47.697862 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:47.697826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc8jl" event={"ID":"51dc6096-8902-4327-b266-0d73cb2f676d","Type":"ContainerStarted","Data":"5c501e0f34526d7461a8d46c3b293c879f62aa276adef9ac73a6014124c3ff23"}
Sep 15 21:10:47.720198 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:47.720143    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rc8jl" podStartSLOduration=9.669991926 podStartE2EDuration="37.720123291s" totalImagesPullingTime="28.050131365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:47.719034743 +0000 UTC m=+37.682092126" watchObservedRunningTime="2026-09-15 21:10:47.720123291 +0000 UTC m=+37.683180666"
Sep 15 21:10:48.113795 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:48.113752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:48.113975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:48.113839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:48.113975 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:48.113898    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:10:48.113975 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:48.113919    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6848588594-gj864: secret "image-registry-tls" not found
Sep 15 21:10:48.113975 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:48.113940    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:10:48.113975 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:48.113973    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls podName:f7835480-8a67-45e3-a4bf-aafaba2af2bb nodeName:}" failed. No retries permitted until 2026-09-15 21:10:52.113958754 +0000 UTC m=+42.077016118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls") pod "image-registry-6848588594-gj864" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb") : secret "image-registry-tls" not found
Sep 15 21:10:48.114219 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:48.113987    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls podName:04d67dbc-688d-42da-a42a-66aa0b7258e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:52.113980335 +0000 UTC m=+42.077037694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls") pod "dns-default-m9jvn" (UID: "04d67dbc-688d-42da-a42a-66aa0b7258e8") : secret "dns-default-metrics-tls" not found
Sep 15 21:10:48.701231 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:48.701201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2gb66" event={"ID":"691cfa1a-d482-4ccb-91d4-e0dc621dc229","Type":"ContainerStarted","Data":"8a761118f637a11212f8ac424c1d6e238b28cd056b9ed7f7c8fa7e2e688cec98"}
Sep 15 21:10:48.718242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:48.718195    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2gb66" podStartSLOduration=2.347419048 podStartE2EDuration="4.718181432s" totalImagesPullingTime="2.370762384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:48.717052723 +0000 UTC m=+38.680110102" watchObservedRunningTime="2026-09-15 21:10:48.718181432 +0000 UTC m=+38.681238815"
Sep 15 21:10:52.142382 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:52.142333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:10:52.142749 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:10:52.142413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:10:52.142749 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:52.142498    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:10:52.142749 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:52.142519    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6848588594-gj864: secret "image-registry-tls" not found
Sep 15 21:10:52.142749 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:52.142527    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:10:52.142749 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:52.142574    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls podName:f7835480-8a67-45e3-a4bf-aafaba2af2bb nodeName:}" failed. No retries permitted until 2026-09-15 21:11:00.142558526 +0000 UTC m=+50.105615890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls") pod "image-registry-6848588594-gj864" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb") : secret "image-registry-tls" not found
Sep 15 21:10:52.142749 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:10:52.142586    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls podName:04d67dbc-688d-42da-a42a-66aa0b7258e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:00.142580525 +0000 UTC m=+50.105637884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls") pod "dns-default-m9jvn" (UID: "04d67dbc-688d-42da-a42a-66aa0b7258e8") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:00.198470 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:00.198432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:11:00.198867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:00.198498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:11:00.198867 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:00.198585    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:00.198867 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:00.198590    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:11:00.198867 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:00.198611    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6848588594-gj864: secret "image-registry-tls" not found
Sep 15 21:11:00.198867 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:00.198639    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls podName:04d67dbc-688d-42da-a42a-66aa0b7258e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:16.198624778 +0000 UTC m=+66.161682137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls") pod "dns-default-m9jvn" (UID: "04d67dbc-688d-42da-a42a-66aa0b7258e8") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:00.198867 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:00.198663    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls podName:f7835480-8a67-45e3-a4bf-aafaba2af2bb nodeName:}" failed. No retries permitted until 2026-09-15 21:11:16.198645733 +0000 UTC m=+66.161703092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls") pod "image-registry-6848588594-gj864" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb") : secret "image-registry-tls" not found
Sep 15 21:11:09.677432 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:09.677402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brhtn"
Sep 15 21:11:16.206796 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.206756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:11:16.206796 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.206795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:11:16.207269 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.206838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:11:16.207269 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:16.206907    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:11:16.207269 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:16.206928    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6848588594-gj864: secret "image-registry-tls" not found
Sep 15 21:11:16.207269 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:16.206937    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:16.207269 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:16.206991    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls podName:f7835480-8a67-45e3-a4bf-aafaba2af2bb nodeName:}" failed. No retries permitted until 2026-09-15 21:11:48.206975252 +0000 UTC m=+98.170032625 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls") pod "image-registry-6848588594-gj864" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb") : secret "image-registry-tls" not found
Sep 15 21:11:16.207269 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:16.207004    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls podName:04d67dbc-688d-42da-a42a-66aa0b7258e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:48.206998589 +0000 UTC m=+98.170055948 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls") pod "dns-default-m9jvn" (UID: "04d67dbc-688d-42da-a42a-66aa0b7258e8") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:16.209187 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.209173    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:11:16.220302 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.220278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e74e7fd1-2672-4c85-b798-a3f036fb7681-original-pull-secret\") pod \"global-pull-secret-syncer-2mthm\" (UID: \"e74e7fd1-2672-4c85-b798-a3f036fb7681\") " pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:11:16.307481 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.307453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdd9\" (UniqueName: \"kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9\") pod \"network-check-target-z477j\" (UID: \"4832498c-7ee0-4464-aae8-919202c79bdd\") " pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:11:16.307648 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.307502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:11:16.309526 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.309505    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:11:16.309627 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.309610    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:11:16.318408 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:16.318387    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 21:11:16.318491 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:16.318442    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs podName:a25ac072-b1cb-4739-a1e7-f0ed49ddf881 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:20.318428352 +0000 UTC m=+130.281485724 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs") pod "network-metrics-daemon-gp2b5" (UID: "a25ac072-b1cb-4739-a1e7-f0ed49ddf881") : secret "metrics-daemon-secret" not found
Sep 15 21:11:16.320125 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.320109    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:11:16.331563 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.331537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdd9\" (UniqueName: \"kubernetes.io/projected/4832498c-7ee0-4464-aae8-919202c79bdd-kube-api-access-cfdd9\") pod \"network-check-target-z477j\" (UID: \"4832498c-7ee0-4464-aae8-919202c79bdd\") " pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:11:16.435722 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.435697    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6cwlg\""
Sep 15 21:11:16.439571 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.439555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2mthm"
Sep 15 21:11:16.439938 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.439911    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:11:16.714693 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.714614    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2mthm"]
Sep 15 21:11:16.717535 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:11:16.717509    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74e7fd1_2672_4c85_b798_a3f036fb7681.slice/crio-8b3e0b96673a1b0b67d82216564e39eee03aa2c484ed5b9f09721fa3eb5412e1 WatchSource:0}: Error finding container 8b3e0b96673a1b0b67d82216564e39eee03aa2c484ed5b9f09721fa3eb5412e1: Status 404 returned error can't find the container with id 8b3e0b96673a1b0b67d82216564e39eee03aa2c484ed5b9f09721fa3eb5412e1
Sep 15 21:11:16.735801 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.735774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z477j"]
Sep 15 21:11:16.738463 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:11:16.738439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4832498c_7ee0_4464_aae8_919202c79bdd.slice/crio-1759edfe766935b399900cec1ce16476fa9cbc4c8bd82ad01f300807762bb431 WatchSource:0}: Error finding container 1759edfe766935b399900cec1ce16476fa9cbc4c8bd82ad01f300807762bb431: Status 404 returned error can't find the container with id 1759edfe766935b399900cec1ce16476fa9cbc4c8bd82ad01f300807762bb431
Sep 15 21:11:16.766019 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.765987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z477j" event={"ID":"4832498c-7ee0-4464-aae8-919202c79bdd","Type":"ContainerStarted","Data":"1759edfe766935b399900cec1ce16476fa9cbc4c8bd82ad01f300807762bb431"}
Sep 15 21:11:16.766971 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:16.766951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2mthm" event={"ID":"e74e7fd1-2672-4c85-b798-a3f036fb7681","Type":"ContainerStarted","Data":"8b3e0b96673a1b0b67d82216564e39eee03aa2c484ed5b9f09721fa3eb5412e1"}
Sep 15 21:11:21.780508 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:21.780472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z477j" event={"ID":"4832498c-7ee0-4464-aae8-919202c79bdd","Type":"ContainerStarted","Data":"02485a9cbe5420c1f722ca8e561c3799b0c208137ff78fd376a4d4f367d30be9"}
Sep 15 21:11:21.781022 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:21.780613    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:11:21.781979 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:21.781955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2mthm" event={"ID":"e74e7fd1-2672-4c85-b798-a3f036fb7681","Type":"ContainerStarted","Data":"213e65677cadc156cfe09ecf7e93f4f024e3e5c1c2414b4efd3049e044be0f35"}
Sep 15 21:11:21.796291 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:21.796253    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z477j" podStartSLOduration=67.433583692 podStartE2EDuration="1m11.796241172s" totalImagesPullingTime="4.36265748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:21.795210806 +0000 UTC m=+71.758268188" watchObservedRunningTime="2026-09-15 21:11:21.796241172 +0000 UTC m=+71.759298553"
Sep 15 21:11:21.809307 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:21.809269    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2mthm" podStartSLOduration=67.42216433 podStartE2EDuration="1m11.809238019s" totalImagesPullingTime="4.387073689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:21.808865036 +0000 UTC m=+71.771922417" watchObservedRunningTime="2026-09-15 21:11:21.809238019 +0000 UTC m=+71.772295400"
Sep 15 21:11:47.233283 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.233251    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-wzf6l"]
Sep 15 21:11:47.235276 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.235261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.239165 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.239141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 15 21:11:47.239304 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.239180    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 21:11:47.239304 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.239207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zlb5z\""
Sep 15 21:11:47.240280 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.240264    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 21:11:47.240713 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.240697    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 15 21:11:47.243120 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.243099    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 15 21:11:47.248220 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.248198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-wzf6l"]
Sep 15 21:11:47.319405 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.319338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-service-ca-bundle\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.319571 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.319420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-tmp\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.319571 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.319480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-snapshots\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.319571 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.319556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.319718 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.319580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-serving-cert\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.319718 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.319602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwg7\" (UniqueName: \"kubernetes.io/projected/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-kube-api-access-7kwg7\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.420805 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.420764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-tmp\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.420984 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.420821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-snapshots\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.420984 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.420861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.420984 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.420876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-serving-cert\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.420984 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.420892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwg7\" (UniqueName: \"kubernetes.io/projected/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-kube-api-access-7kwg7\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.420984 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.420921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-service-ca-bundle\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.421250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.421227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-tmp\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.421681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.421652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-snapshots\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.421944 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.421925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-service-ca-bundle\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.422247 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.422229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.423109 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.423091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-serving-cert\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.429338 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.429317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwg7\" (UniqueName: \"kubernetes.io/projected/fd1f9f59-883a-4b1c-80af-41e5ebc914f0-kube-api-access-7kwg7\") pod \"insights-operator-5844b856db-wzf6l\" (UID: \"fd1f9f59-883a-4b1c-80af-41e5ebc914f0\") " pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.544068 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.543969    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-wzf6l"
Sep 15 21:11:47.810141 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.810055    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-wzf6l"]
Sep 15 21:11:47.813899 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:11:47.813873    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1f9f59_883a_4b1c_80af_41e5ebc914f0.slice/crio-8ada38f682113d6c7f796daa6cf54ddf673d9b686552ea04be68362b68f6f102 WatchSource:0}: Error finding container 8ada38f682113d6c7f796daa6cf54ddf673d9b686552ea04be68362b68f6f102: Status 404 returned error can't find the container with id 8ada38f682113d6c7f796daa6cf54ddf673d9b686552ea04be68362b68f6f102
Sep 15 21:11:47.844135 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:47.844103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wzf6l" event={"ID":"fd1f9f59-883a-4b1c-80af-41e5ebc914f0","Type":"ContainerStarted","Data":"8ada38f682113d6c7f796daa6cf54ddf673d9b686552ea04be68362b68f6f102"}
Sep 15 21:11:48.227877 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:48.227839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls\") pod \"image-registry-6848588594-gj864\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") " pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:11:48.228085 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:48.227891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:11:48.228085 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:48.228024    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 21:11:48.228085 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:48.228047    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6848588594-gj864: secret "image-registry-tls" not found
Sep 15 21:11:48.228085 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:48.228065    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:48.228277 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:48.228128    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls podName:04d67dbc-688d-42da-a42a-66aa0b7258e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:52.228112181 +0000 UTC m=+162.191169552 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls") pod "dns-default-m9jvn" (UID: "04d67dbc-688d-42da-a42a-66aa0b7258e8") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:48.228277 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:48.228141    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls podName:f7835480-8a67-45e3-a4bf-aafaba2af2bb nodeName:}" failed. No retries permitted until 2026-09-15 21:12:52.228135782 +0000 UTC m=+162.191193141 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls") pod "image-registry-6848588594-gj864" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb") : secret "image-registry-tls" not found
Sep 15 21:11:49.851388 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:49.851331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wzf6l" event={"ID":"fd1f9f59-883a-4b1c-80af-41e5ebc914f0","Type":"ContainerStarted","Data":"a9e84a5f9180700faad9b0bb57f172320e2d7d0589e58bd08892164f5a167cad"}
Sep 15 21:11:49.870239 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:49.870184    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-wzf6l" podStartSLOduration=0.99421403 podStartE2EDuration="2.870140864s" totalImagesPullingTime="1.875926834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:49.869719133 +0000 UTC m=+99.832776513" watchObservedRunningTime="2026-09-15 21:11:49.870140864 +0000 UTC m=+99.833198247"
Sep 15 21:11:50.645947 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.645916    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cv55t"]
Sep 15 21:11:50.648199 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.648176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.650481 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.650461    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 21:11:50.650678 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.650660    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qznqd\""
Sep 15 21:11:50.650737 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.650695    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 21:11:50.661618 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.661593    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cv55t"]
Sep 15 21:11:50.749493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.749466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.749493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.749495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-data-volume\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.749665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.749524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-crio-socket\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.749665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.749543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.749665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.749579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6sw\" (UniqueName: \"kubernetes.io/projected/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-api-access-dt6sw\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.850856 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.850827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-data-volume\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.850991 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.850872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-crio-socket\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.850991 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.850893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.850991 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.850911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6sw\" (UniqueName: \"kubernetes.io/projected/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-api-access-dt6sw\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.850991 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.850965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-crio-socket\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.850991 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:50.850987    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:50.851184 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:50.851042    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:11:51.351027501 +0000 UTC m=+101.314084860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:50.851184 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:50.851042    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:11:50.851184 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.850973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.851184 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:50.851099    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:11:51.351083882 +0000 UTC m=+101.314141264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : configmap "kube-rbac-proxy" not found
Sep 15 21:11:50.851326 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.851247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-data-volume\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:50.860038 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:50.860012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6sw\" (UniqueName: \"kubernetes.io/projected/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-api-access-dt6sw\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:51.355046 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:51.355007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:51.355213 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:51.355061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:51.355213 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:51.355131    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:11:51.355213 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:51.355173    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:51.355213 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:51.355198    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:11:52.355181768 +0000 UTC m=+102.318239144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : configmap "kube-rbac-proxy" not found
Sep 15 21:11:51.355213 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:51.355215    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:11:52.355208481 +0000 UTC m=+102.318265840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:52.363362 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:52.363320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:52.363717 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:52.363487    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:52.363717 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:52.363527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:52.363717 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:52.363549    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:11:54.363534567 +0000 UTC m=+104.326591939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:52.363717 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:52.363584    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:11:52.363717 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:52.363623    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:11:54.363612019 +0000 UTC m=+104.326669393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : configmap "kube-rbac-proxy" not found
Sep 15 21:11:52.787239 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:52.787203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z477j"
Sep 15 21:11:52.788036 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:52.788018    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sds6c_8060115d-0d03-495c-bd01-c0a4315c501f/dns-node-resolver/0.log"
Sep 15 21:11:53.388361 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:53.388317    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hcnhm_61e8e166-9aa3-4f53-b9d2-564ce990235c/node-ca/0.log"
Sep 15 21:11:54.188185 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:54.188154    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2gb66_691cfa1a-d482-4ccb-91d4-e0dc621dc229/serve-healthcheck-canary/0.log"
Sep 15 21:11:54.380712 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:54.380681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:54.380860 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:54.380752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:54.380860 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:54.380820    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:11:54.380962 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:54.380871    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:11:58.380859387 +0000 UTC m=+108.343916746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : configmap "kube-rbac-proxy" not found
Sep 15 21:11:54.380962 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:54.380875    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:54.380962 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:54.380943    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:11:58.380924332 +0000 UTC m=+108.343981711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:57.291611 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.291575    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"]
Sep 15 21:11:57.294323 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.294307    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"]
Sep 15 21:11:57.294453 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.294436    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.299081 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.299041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.299246 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.299057    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 15 21:11:57.299310 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.299257    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:11:57.299926 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.299257    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 15 21:11:57.299926 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.299903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hl9v5\""
Sep 15 21:11:57.300245 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.300227    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 15 21:11:57.301947 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.301928    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4zpk2\""
Sep 15 21:11:57.302079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.302058    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:11:57.302388 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.302370    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 15 21:11:57.303980 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.303768    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 15 21:11:57.304142 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.304120    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 15 21:11:57.305137 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.305103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"]
Sep 15 21:11:57.306034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.306017    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"]
Sep 15 21:11:57.405230 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.405194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37eb94cb-0bca-47f2-bcc0-7bcda93efd65-serving-cert\") pod \"service-ca-operator-cf7c776b7-qzndj\" (UID: \"37eb94cb-0bca-47f2-bcc0-7bcda93efd65\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.405230 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.405229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzxg\" (UniqueName: \"kubernetes.io/projected/37eb94cb-0bca-47f2-bcc0-7bcda93efd65-kube-api-access-cmzxg\") pod \"service-ca-operator-cf7c776b7-qzndj\" (UID: \"37eb94cb-0bca-47f2-bcc0-7bcda93efd65\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.405521 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.405268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e7474e7-64cf-464e-b7f4-d8114506b756-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-q6lc7\" (UID: \"5e7474e7-64cf-464e-b7f4-d8114506b756\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.405521 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.405335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37eb94cb-0bca-47f2-bcc0-7bcda93efd65-config\") pod \"service-ca-operator-cf7c776b7-qzndj\" (UID: \"37eb94cb-0bca-47f2-bcc0-7bcda93efd65\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.405521 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.405398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e7474e7-64cf-464e-b7f4-d8114506b756-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-q6lc7\" (UID: \"5e7474e7-64cf-464e-b7f4-d8114506b756\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.405521 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.405429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8brb\" (UniqueName: \"kubernetes.io/projected/5e7474e7-64cf-464e-b7f4-d8114506b756-kube-api-access-l8brb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-q6lc7\" (UID: \"5e7474e7-64cf-464e-b7f4-d8114506b756\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.506516 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.506484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37eb94cb-0bca-47f2-bcc0-7bcda93efd65-serving-cert\") pod \"service-ca-operator-cf7c776b7-qzndj\" (UID: \"37eb94cb-0bca-47f2-bcc0-7bcda93efd65\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.506646 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.506521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzxg\" (UniqueName: \"kubernetes.io/projected/37eb94cb-0bca-47f2-bcc0-7bcda93efd65-kube-api-access-cmzxg\") pod \"service-ca-operator-cf7c776b7-qzndj\" (UID: \"37eb94cb-0bca-47f2-bcc0-7bcda93efd65\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.506708 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.506651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e7474e7-64cf-464e-b7f4-d8114506b756-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-q6lc7\" (UID: \"5e7474e7-64cf-464e-b7f4-d8114506b756\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.506708 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.506692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37eb94cb-0bca-47f2-bcc0-7bcda93efd65-config\") pod \"service-ca-operator-cf7c776b7-qzndj\" (UID: \"37eb94cb-0bca-47f2-bcc0-7bcda93efd65\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.506813 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.506728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e7474e7-64cf-464e-b7f4-d8114506b756-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-q6lc7\" (UID: \"5e7474e7-64cf-464e-b7f4-d8114506b756\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.506813 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.506761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8brb\" (UniqueName: \"kubernetes.io/projected/5e7474e7-64cf-464e-b7f4-d8114506b756-kube-api-access-l8brb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-q6lc7\" (UID: \"5e7474e7-64cf-464e-b7f4-d8114506b756\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.507258 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.507235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37eb94cb-0bca-47f2-bcc0-7bcda93efd65-config\") pod \"service-ca-operator-cf7c776b7-qzndj\" (UID: \"37eb94cb-0bca-47f2-bcc0-7bcda93efd65\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.507420 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.507259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e7474e7-64cf-464e-b7f4-d8114506b756-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-q6lc7\" (UID: \"5e7474e7-64cf-464e-b7f4-d8114506b756\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.508753 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.508732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37eb94cb-0bca-47f2-bcc0-7bcda93efd65-serving-cert\") pod \"service-ca-operator-cf7c776b7-qzndj\" (UID: \"37eb94cb-0bca-47f2-bcc0-7bcda93efd65\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.508886 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.508869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e7474e7-64cf-464e-b7f4-d8114506b756-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-q6lc7\" (UID: \"5e7474e7-64cf-464e-b7f4-d8114506b756\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.515023 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.514997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8brb\" (UniqueName: \"kubernetes.io/projected/5e7474e7-64cf-464e-b7f4-d8114506b756-kube-api-access-l8brb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-q6lc7\" (UID: \"5e7474e7-64cf-464e-b7f4-d8114506b756\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.515211 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.515195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzxg\" (UniqueName: \"kubernetes.io/projected/37eb94cb-0bca-47f2-bcc0-7bcda93efd65-kube-api-access-cmzxg\") pod \"service-ca-operator-cf7c776b7-qzndj\" (UID: \"37eb94cb-0bca-47f2-bcc0-7bcda93efd65\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.606419 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.606356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"
Sep 15 21:11:57.611702 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.611686    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"
Sep 15 21:11:57.874128 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.874102    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7"]
Sep 15 21:11:57.877391 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:11:57.877363    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7474e7_64cf_464e_b7f4_d8114506b756.slice/crio-27267f3ff719f68bb69f087cf2aceb554a174a1e83ee3afa07436a368ed5e9b1 WatchSource:0}: Error finding container 27267f3ff719f68bb69f087cf2aceb554a174a1e83ee3afa07436a368ed5e9b1: Status 404 returned error can't find the container with id 27267f3ff719f68bb69f087cf2aceb554a174a1e83ee3afa07436a368ed5e9b1
Sep 15 21:11:57.879473 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:57.879447    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj"]
Sep 15 21:11:57.881739 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:11:57.881713    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37eb94cb_0bca_47f2_bcc0_7bcda93efd65.slice/crio-c9c112ade4d500cd9f9e665b04086855f86dd4b360b8f29a610ae340f6932213 WatchSource:0}: Error finding container c9c112ade4d500cd9f9e665b04086855f86dd4b360b8f29a610ae340f6932213: Status 404 returned error can't find the container with id c9c112ade4d500cd9f9e665b04086855f86dd4b360b8f29a610ae340f6932213
Sep 15 21:11:58.414635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:58.414594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:58.415067 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:58.414669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:11:58.415067 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:58.414692    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:11:58.415067 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:58.414775    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:12:06.414751415 +0000 UTC m=+116.377808795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : configmap "kube-rbac-proxy" not found
Sep 15 21:11:58.415067 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:58.414831    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:58.415067 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:11:58.414891    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:12:06.414874448 +0000 UTC m=+116.377931808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:58.875297 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:58.875263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7" event={"ID":"5e7474e7-64cf-464e-b7f4-d8114506b756","Type":"ContainerStarted","Data":"27267f3ff719f68bb69f087cf2aceb554a174a1e83ee3afa07436a368ed5e9b1"}
Sep 15 21:11:58.876647 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:11:58.876616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj" event={"ID":"37eb94cb-0bca-47f2-bcc0-7bcda93efd65","Type":"ContainerStarted","Data":"c9c112ade4d500cd9f9e665b04086855f86dd4b360b8f29a610ae340f6932213"}
Sep 15 21:12:00.885208 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:00.885162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj" event={"ID":"37eb94cb-0bca-47f2-bcc0-7bcda93efd65","Type":"ContainerStarted","Data":"55118f112c1551b105b331704552d178231157167325be9793a2b81600cc3888"}
Sep 15 21:12:00.886764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:00.886738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7" event={"ID":"5e7474e7-64cf-464e-b7f4-d8114506b756","Type":"ContainerStarted","Data":"621dfac6363467bad6086fe6989bee31943fc98ff9f1c20d8d4b85bba30339ca"}
Sep 15 21:12:00.901419 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:00.901374    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj" podStartSLOduration=1.51653227 podStartE2EDuration="3.901341717s" totalImagesPullingTime="2.384809447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:00.899898057 +0000 UTC m=+110.862955438" watchObservedRunningTime="2026-09-15 21:12:00.901341717 +0000 UTC m=+110.864399099"
Sep 15 21:12:00.916778 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:00.916730    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7" podStartSLOduration=1.52720057 podStartE2EDuration="3.916711818s" totalImagesPullingTime="2.389511248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:00.915617995 +0000 UTC m=+110.878675376" watchObservedRunningTime="2026-09-15 21:12:00.916711818 +0000 UTC m=+110.879769200"
Sep 15 21:12:02.042788 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.042753    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst"]
Sep 15 21:12:02.044658 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.044643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst"
Sep 15 21:12:02.047021 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.046996    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 21:12:02.047131 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.046997    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 21:12:02.047131 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.046997    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-l6g94\""
Sep 15 21:12:02.052236 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.052215    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst"]
Sep 15 21:12:02.150541 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.150505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdd9s\" (UniqueName: \"kubernetes.io/projected/abd57f26-b082-4f2f-af4e-b444ff67a234-kube-api-access-pdd9s\") pod \"migrator-8b4d697c5-zbsst\" (UID: \"abd57f26-b082-4f2f-af4e-b444ff67a234\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst"
Sep 15 21:12:02.251837 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.251791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdd9s\" (UniqueName: \"kubernetes.io/projected/abd57f26-b082-4f2f-af4e-b444ff67a234-kube-api-access-pdd9s\") pod \"migrator-8b4d697c5-zbsst\" (UID: \"abd57f26-b082-4f2f-af4e-b444ff67a234\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst"
Sep 15 21:12:02.260946 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.260919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdd9s\" (UniqueName: \"kubernetes.io/projected/abd57f26-b082-4f2f-af4e-b444ff67a234-kube-api-access-pdd9s\") pod \"migrator-8b4d697c5-zbsst\" (UID: \"abd57f26-b082-4f2f-af4e-b444ff67a234\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst"
Sep 15 21:12:02.353372 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.353242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst"
Sep 15 21:12:02.614979 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.614912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst"]
Sep 15 21:12:02.617735 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:02.617708    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd57f26_b082_4f2f_af4e_b444ff67a234.slice/crio-27800a04ca02e14e86056cc6419badaa07b972a88a9edf83326e2d8cc839656b WatchSource:0}: Error finding container 27800a04ca02e14e86056cc6419badaa07b972a88a9edf83326e2d8cc839656b: Status 404 returned error can't find the container with id 27800a04ca02e14e86056cc6419badaa07b972a88a9edf83326e2d8cc839656b
Sep 15 21:12:02.894486 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:02.894452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst" event={"ID":"abd57f26-b082-4f2f-af4e-b444ff67a234","Type":"ContainerStarted","Data":"27800a04ca02e14e86056cc6419badaa07b972a88a9edf83326e2d8cc839656b"}
Sep 15 21:12:03.899244 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:03.899208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst" event={"ID":"abd57f26-b082-4f2f-af4e-b444ff67a234","Type":"ContainerStarted","Data":"1f826222fc2b444355406f6cf0a89cecb880b59084a33568f1249262d613e6e4"}
Sep 15 21:12:03.899614 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:03.899286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst" event={"ID":"abd57f26-b082-4f2f-af4e-b444ff67a234","Type":"ContainerStarted","Data":"c103912639c57628e1867f3ab16d2960a41073ddfd489325921fad3eb85e300d"}
Sep 15 21:12:03.915714 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:03.915618    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-zbsst" podStartSLOduration=0.877460738 podStartE2EDuration="1.915604487s" totalImagesPullingTime="1.038143749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:03.915290037 +0000 UTC m=+113.878347418" watchObservedRunningTime="2026-09-15 21:12:03.915604487 +0000 UTC m=+113.878661867"
Sep 15 21:12:06.484994 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:06.484955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:12:06.485372 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:12:06.485117    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:06.485372 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:06.485124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:12:06.485372 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:12:06.485181    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls podName:eb1e5c25-047e-44b4-b262-c21e8f1a2f3c nodeName:}" failed. No retries permitted until 2026-09-15 21:12:22.485165282 +0000 UTC m=+132.448222654 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv55t" (UID: "eb1e5c25-047e-44b4-b262-c21e8f1a2f3c") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:06.485590 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:06.485574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:12:20.387837 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:20.387803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:12:20.390052 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:20.390027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25ac072-b1cb-4739-a1e7-f0ed49ddf881-metrics-certs\") pod \"network-metrics-daemon-gp2b5\" (UID: \"a25ac072-b1cb-4739-a1e7-f0ed49ddf881\") " pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:12:20.641095 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:20.641014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8fnh6\""
Sep 15 21:12:20.646301 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:20.646285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gp2b5"
Sep 15 21:12:20.909370 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:20.909281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gp2b5"]
Sep 15 21:12:20.912052 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:20.912025    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25ac072_b1cb_4739_a1e7_f0ed49ddf881.slice/crio-c523f2f0c9b79fd4e259e67c56ab4e6287cf7b3766cc02c45ceaa60d3030b3a5 WatchSource:0}: Error finding container c523f2f0c9b79fd4e259e67c56ab4e6287cf7b3766cc02c45ceaa60d3030b3a5: Status 404 returned error can't find the container with id c523f2f0c9b79fd4e259e67c56ab4e6287cf7b3766cc02c45ceaa60d3030b3a5
Sep 15 21:12:20.950429 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:20.950400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gp2b5" event={"ID":"a25ac072-b1cb-4739-a1e7-f0ed49ddf881","Type":"ContainerStarted","Data":"c523f2f0c9b79fd4e259e67c56ab4e6287cf7b3766cc02c45ceaa60d3030b3a5"}
Sep 15 21:12:22.507063 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:22.507026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:12:22.509215 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:22.509197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1e5c25-047e-44b4-b262-c21e8f1a2f3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv55t\" (UID: \"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c\") " pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:12:22.758467 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:22.758413    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qznqd\""
Sep 15 21:12:22.767183 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:22.767163    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cv55t"
Sep 15 21:12:22.960065 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:22.960038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gp2b5" event={"ID":"a25ac072-b1cb-4739-a1e7-f0ed49ddf881","Type":"ContainerStarted","Data":"be776b5136a239be69b234f54b729929b33365447d8de31595a5fa7c5ebd5e84"}
Sep 15 21:12:22.960193 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:22.960083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gp2b5" event={"ID":"a25ac072-b1cb-4739-a1e7-f0ed49ddf881","Type":"ContainerStarted","Data":"f75bedb023862c365b9906225a5a2c1eef0617084eb7ad64d74ffbe55a4d4ff2"}
Sep 15 21:12:22.976776 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:22.976735    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gp2b5" podStartSLOduration=131.974137849 podStartE2EDuration="2m12.976721489s" totalImagesPullingTime="1.00258364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:22.975658399 +0000 UTC m=+132.938715779" watchObservedRunningTime="2026-09-15 21:12:22.976721489 +0000 UTC m=+132.939778870"
Sep 15 21:12:23.034159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:23.033862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cv55t"]
Sep 15 21:12:23.036292 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:23.036270    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1e5c25_047e_44b4_b262_c21e8f1a2f3c.slice/crio-3a791839069dd555672074e8ae2592586519ac913c18373672ba5c3e52707a7f WatchSource:0}: Error finding container 3a791839069dd555672074e8ae2592586519ac913c18373672ba5c3e52707a7f: Status 404 returned error can't find the container with id 3a791839069dd555672074e8ae2592586519ac913c18373672ba5c3e52707a7f
Sep 15 21:12:23.964537 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:23.964473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cv55t" event={"ID":"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c","Type":"ContainerStarted","Data":"67c0bc2e211a28845b6511183b977eec8933b69e335a12544c116be91414ae41"}
Sep 15 21:12:23.964537 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:23.964521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cv55t" event={"ID":"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c","Type":"ContainerStarted","Data":"2dc648bfa5fbff4de3052c5779633cb890ce0b1b04c412df92f43c9fd2ad7745"}
Sep 15 21:12:23.964878 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:23.964541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cv55t" event={"ID":"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c","Type":"ContainerStarted","Data":"3a791839069dd555672074e8ae2592586519ac913c18373672ba5c3e52707a7f"}
Sep 15 21:12:25.971827 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:25.971791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cv55t" event={"ID":"eb1e5c25-047e-44b4-b262-c21e8f1a2f3c","Type":"ContainerStarted","Data":"f22266ccdd40632dc9a70d33b5b451b1ee447e90ed1853f4a12dfc14657747ef"}
Sep 15 21:12:25.989815 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:25.989770    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cv55t" podStartSLOduration=33.755420047 podStartE2EDuration="35.989756559s" totalImagesPullingTime="2.234336512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:11:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:25.988151816 +0000 UTC m=+135.951209196" watchObservedRunningTime="2026-09-15 21:12:25.989756559 +0000 UTC m=+135.952813938"
Sep 15 21:12:28.446621 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.446588    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh"]
Sep 15 21:12:28.449855 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.449835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh"
Sep 15 21:12:28.452620 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.452603    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qd7xn\""
Sep 15 21:12:28.465037 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.465001    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"]
Sep 15 21:12:28.468184 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.468163    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh"]
Sep 15 21:12:28.468304 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.468273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"
Sep 15 21:12:28.470479 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.470454    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 21:12:28.470567 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.470484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8r5p6\""
Sep 15 21:12:28.470640 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.470610    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 21:12:28.475256 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.475235    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6848588594-gj864"]
Sep 15 21:12:28.475462 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:12:28.475442    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6848588594-gj864" podUID="f7835480-8a67-45e3-a4bf-aafaba2af2bb"
Sep 15 21:12:28.480654 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.480635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"]
Sep 15 21:12:28.549564 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.549538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvbk\" (UniqueName: \"kubernetes.io/projected/35facb96-95a4-479f-9f38-4617a494db9e-kube-api-access-whvbk\") pod \"network-check-source-6f8b768455-dg8qh\" (UID: \"35facb96-95a4-479f-9f38-4617a494db9e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh"
Sep 15 21:12:28.549564 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.549568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28e7256c-de77-4363-9395-2004b91d28f6-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-r9l8p\" (UID: \"28e7256c-de77-4363-9395-2004b91d28f6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"
Sep 15 21:12:28.549762 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.549602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/28e7256c-de77-4363-9395-2004b91d28f6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r9l8p\" (UID: \"28e7256c-de77-4363-9395-2004b91d28f6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"
Sep 15 21:12:28.556847 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.556814    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649"]
Sep 15 21:12:28.559881 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.559860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649"
Sep 15 21:12:28.562293 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.562273    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 15 21:12:28.562396 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.562329    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5xcst\""
Sep 15 21:12:28.574432 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.574411    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649"]
Sep 15 21:12:28.650645 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.650616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvbk\" (UniqueName: \"kubernetes.io/projected/35facb96-95a4-479f-9f38-4617a494db9e-kube-api-access-whvbk\") pod \"network-check-source-6f8b768455-dg8qh\" (UID: \"35facb96-95a4-479f-9f38-4617a494db9e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh"
Sep 15 21:12:28.650645 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.650648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28e7256c-de77-4363-9395-2004b91d28f6-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-r9l8p\" (UID: \"28e7256c-de77-4363-9395-2004b91d28f6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"
Sep 15 21:12:28.650854 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.650675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5514f3e1-2f67-4d32-bce2-72f7de833968-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j5649\" (UID: \"5514f3e1-2f67-4d32-bce2-72f7de833968\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649"
Sep 15 21:12:28.650854 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.650717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/28e7256c-de77-4363-9395-2004b91d28f6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r9l8p\" (UID: \"28e7256c-de77-4363-9395-2004b91d28f6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"
Sep 15 21:12:28.651429 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.651404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28e7256c-de77-4363-9395-2004b91d28f6-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-r9l8p\" (UID: \"28e7256c-de77-4363-9395-2004b91d28f6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"
Sep 15 21:12:28.653136 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.653116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/28e7256c-de77-4363-9395-2004b91d28f6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r9l8p\" (UID: \"28e7256c-de77-4363-9395-2004b91d28f6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"
Sep 15 21:12:28.659952 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.659929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvbk\" (UniqueName: \"kubernetes.io/projected/35facb96-95a4-479f-9f38-4617a494db9e-kube-api-access-whvbk\") pod \"network-check-source-6f8b768455-dg8qh\" (UID: \"35facb96-95a4-479f-9f38-4617a494db9e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh"
Sep 15 21:12:28.751500 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.751420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5514f3e1-2f67-4d32-bce2-72f7de833968-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j5649\" (UID: \"5514f3e1-2f67-4d32-bce2-72f7de833968\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649"
Sep 15 21:12:28.753885 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.753863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5514f3e1-2f67-4d32-bce2-72f7de833968-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j5649\" (UID: \"5514f3e1-2f67-4d32-bce2-72f7de833968\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649"
Sep 15 21:12:28.761216 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.761198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh"
Sep 15 21:12:28.776454 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.776429    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"
Sep 15 21:12:28.868471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.868440    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649"
Sep 15 21:12:28.980977 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.980943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:12:28.988449 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:28.988427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:12:29.031506 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.031421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh"]
Sep 15 21:12:29.034978 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:29.034946    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35facb96_95a4_479f_9f38_4617a494db9e.slice/crio-0054c29140155e21a089d986e3caa3904eaa5629637ca90cce2185129c164615 WatchSource:0}: Error finding container 0054c29140155e21a089d986e3caa3904eaa5629637ca90cce2185129c164615: Status 404 returned error can't find the container with id 0054c29140155e21a089d986e3caa3904eaa5629637ca90cce2185129c164615
Sep 15 21:12:29.052279 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.052253    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p"]
Sep 15 21:12:29.054260 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.054241    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-installation-pull-secrets\") pod \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") "
Sep 15 21:12:29.054332 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.054270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-certificates\") pod \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") "
Sep 15 21:12:29.054332 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.054313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-image-registry-private-configuration\") pod \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") "
Sep 15 21:12:29.054452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.054335    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7835480-8a67-45e3-a4bf-aafaba2af2bb-ca-trust-extracted\") pod \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") "
Sep 15 21:12:29.054452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.054383    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jdgh\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-kube-api-access-4jdgh\") pod \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") "
Sep 15 21:12:29.054452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.054421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-bound-sa-token\") pod \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") "
Sep 15 21:12:29.054596 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.054459    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-trusted-ca\") pod \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\" (UID: \"f7835480-8a67-45e3-a4bf-aafaba2af2bb\") "
Sep 15 21:12:29.054678 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.054648    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7835480-8a67-45e3-a4bf-aafaba2af2bb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f7835480-8a67-45e3-a4bf-aafaba2af2bb" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:12:29.054745 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.054701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f7835480-8a67-45e3-a4bf-aafaba2af2bb" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:12:29.054944 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.054926    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f7835480-8a67-45e3-a4bf-aafaba2af2bb" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:12:29.056387 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.056359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f7835480-8a67-45e3-a4bf-aafaba2af2bb" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:12:29.056558 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.056539    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f7835480-8a67-45e3-a4bf-aafaba2af2bb" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:12:29.056741 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.056721    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f7835480-8a67-45e3-a4bf-aafaba2af2bb" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:12:29.056785 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.056751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-kube-api-access-4jdgh" (OuterVolumeSpecName: "kube-api-access-4jdgh") pod "f7835480-8a67-45e3-a4bf-aafaba2af2bb" (UID: "f7835480-8a67-45e3-a4bf-aafaba2af2bb"). InnerVolumeSpecName "kube-api-access-4jdgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:12:29.062354 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:29.062320    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e7256c_de77_4363_9395_2004b91d28f6.slice/crio-f45412d54c1c4464f1503544fedf26de00bec3a77a95e6a473a5dbbd1bd3175a WatchSource:0}: Error finding container f45412d54c1c4464f1503544fedf26de00bec3a77a95e6a473a5dbbd1bd3175a: Status 404 returned error can't find the container with id f45412d54c1c4464f1503544fedf26de00bec3a77a95e6a473a5dbbd1bd3175a
Sep 15 21:12:29.155049 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.155015    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-bound-sa-token\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:12:29.155049 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.155042    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-trusted-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:12:29.155049 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.155056    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-installation-pull-secrets\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:12:29.155489 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.155068    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-certificates\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:12:29.155489 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.155077    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7835480-8a67-45e3-a4bf-aafaba2af2bb-image-registry-private-configuration\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:12:29.155489 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.155086    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7835480-8a67-45e3-a4bf-aafaba2af2bb-ca-trust-extracted\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:12:29.155489 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.155099    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdgh\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-kube-api-access-4jdgh\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:12:29.156780 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.156753    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649"]
Sep 15 21:12:29.159266 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:29.159241    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5514f3e1_2f67_4d32_bce2_72f7de833968.slice/crio-a3de7c8d4e81ffb23b900e7752ce0ed0cb6a213ee244761aca9dff575d596465 WatchSource:0}: Error finding container a3de7c8d4e81ffb23b900e7752ce0ed0cb6a213ee244761aca9dff575d596465: Status 404 returned error can't find the container with id a3de7c8d4e81ffb23b900e7752ce0ed0cb6a213ee244761aca9dff575d596465
Sep 15 21:12:29.987857 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.987774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p" event={"ID":"28e7256c-de77-4363-9395-2004b91d28f6","Type":"ContainerStarted","Data":"f45412d54c1c4464f1503544fedf26de00bec3a77a95e6a473a5dbbd1bd3175a"}
Sep 15 21:12:29.989089 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.989056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649" event={"ID":"5514f3e1-2f67-4d32-bce2-72f7de833968","Type":"ContainerStarted","Data":"a3de7c8d4e81ffb23b900e7752ce0ed0cb6a213ee244761aca9dff575d596465"}
Sep 15 21:12:29.990772 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.990746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh" event={"ID":"35facb96-95a4-479f-9f38-4617a494db9e","Type":"ContainerStarted","Data":"2e76f9f8b0a37cd207afaa8be8a0686c57bbcfe38b3d9fdc9ef8ee0607f1636b"}
Sep 15 21:12:29.990878 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.990793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh" event={"ID":"35facb96-95a4-479f-9f38-4617a494db9e","Type":"ContainerStarted","Data":"0054c29140155e21a089d986e3caa3904eaa5629637ca90cce2185129c164615"}
Sep 15 21:12:29.990878 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:29.990756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6848588594-gj864"
Sep 15 21:12:30.007576 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:30.007530    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dg8qh" podStartSLOduration=2.007514414 podStartE2EDuration="2.007514414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:30.005695028 +0000 UTC m=+139.968752409" watchObservedRunningTime="2026-09-15 21:12:30.007514414 +0000 UTC m=+139.970571793"
Sep 15 21:12:30.036093 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:30.036054    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6848588594-gj864"]
Sep 15 21:12:30.040022 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:30.039998    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6848588594-gj864"]
Sep 15 21:12:30.163606 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:30.163569    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7835480-8a67-45e3-a4bf-aafaba2af2bb-registry-tls\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:12:30.524852 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:30.524772    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7835480-8a67-45e3-a4bf-aafaba2af2bb" path="/var/lib/kubelet/pods/f7835480-8a67-45e3-a4bf-aafaba2af2bb/volumes"
Sep 15 21:12:30.996249 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:30.996212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p" event={"ID":"28e7256c-de77-4363-9395-2004b91d28f6","Type":"ContainerStarted","Data":"a13715a72581086bf76f23ecef50d8cbf42f051d02ae099fbf766e57127c02c6"}
Sep 15 21:12:31.015281 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:31.015148    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r9l8p" podStartSLOduration=1.835207788 podStartE2EDuration="3.015130011s" totalImagesPullingTime="1.179922223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:31.014096173 +0000 UTC m=+140.977153576" watchObservedRunningTime="2026-09-15 21:12:31.015130011 +0000 UTC m=+140.978187392"
Sep 15 21:12:32.001059 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:32.001021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649" event={"ID":"5514f3e1-2f67-4d32-bce2-72f7de833968","Type":"ContainerStarted","Data":"53218fb579bb0cadebab056e4d76e871dfdb634fec518569cc615b3ef595d871"}
Sep 15 21:12:32.016892 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:32.016831    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649" podStartSLOduration=2.094395241 podStartE2EDuration="4.016816726s" totalImagesPullingTime="1.922421485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:32.015417535 +0000 UTC m=+141.978474912" watchObservedRunningTime="2026-09-15 21:12:32.016816726 +0000 UTC m=+141.979874135"
Sep 15 21:12:33.003968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.003937    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649"
Sep 15 21:12:33.008583 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.008556    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j5649"
Sep 15 21:12:33.787978 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.787943    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"]
Sep 15 21:12:33.790913 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.790897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.793142 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.793120    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 21:12:33.793246 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.793119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 21:12:33.793975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.793954    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5r4kl\""
Sep 15 21:12:33.794070 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.793974    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 21:12:33.794070 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.793989    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 21:12:33.794070 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.793979    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 21:12:33.800882 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.800862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"]
Sep 15 21:12:33.892624 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.892585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.892777 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.892638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.892777 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.892718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.892777 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.892765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nllvh\" (UniqueName: \"kubernetes.io/projected/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-kube-api-access-nllvh\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.993872 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.993832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.994039 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.993885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.994039 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.993915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.994039 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.993948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nllvh\" (UniqueName: \"kubernetes.io/projected/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-kube-api-access-nllvh\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.994551 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.994519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.996356 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.996326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:33.996452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:33.996432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:34.003326 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:34.003300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nllvh\" (UniqueName: \"kubernetes.io/projected/dd2f243b-db86-4a5a-b0ae-820c1e4869ee-kube-api-access-nllvh\") pod \"prometheus-operator-68bcd7f987-tlkbf\" (UID: \"dd2f243b-db86-4a5a-b0ae-820c1e4869ee\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:34.100451 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:34.100371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"
Sep 15 21:12:34.369047 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:34.368975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf"]
Sep 15 21:12:34.372071 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:34.372042    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2f243b_db86_4a5a_b0ae_820c1e4869ee.slice/crio-fc51074631a384131f599a6440ac4d8a78fd61fdc3394d668194824f7f3c806e WatchSource:0}: Error finding container fc51074631a384131f599a6440ac4d8a78fd61fdc3394d668194824f7f3c806e: Status 404 returned error can't find the container with id fc51074631a384131f599a6440ac4d8a78fd61fdc3394d668194824f7f3c806e
Sep 15 21:12:35.010680 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:35.010644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf" event={"ID":"dd2f243b-db86-4a5a-b0ae-820c1e4869ee","Type":"ContainerStarted","Data":"fc51074631a384131f599a6440ac4d8a78fd61fdc3394d668194824f7f3c806e"}
Sep 15 21:12:36.016309 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:36.016274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf" event={"ID":"dd2f243b-db86-4a5a-b0ae-820c1e4869ee","Type":"ContainerStarted","Data":"274ff7948a30f6a9c63123536380e9b29e36ff6d4c6ecd201dbb6e30e5b760d8"}
Sep 15 21:12:36.016703 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:36.016336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf" event={"ID":"dd2f243b-db86-4a5a-b0ae-820c1e4869ee","Type":"ContainerStarted","Data":"acca1b0579a74980aa0a7dc62bdba8adcf1758ab682ef639caf99de9fc542497"}
Sep 15 21:12:36.034819 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:36.034747    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tlkbf" podStartSLOduration=1.544826355 podStartE2EDuration="3.034730588s" totalImagesPullingTime="1.489904233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:36.03284791 +0000 UTC m=+145.995905302" watchObservedRunningTime="2026-09-15 21:12:36.034730588 +0000 UTC m=+145.997788017"
Sep 15 21:12:38.188768 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.188722    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"]
Sep 15 21:12:38.193138 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.193122    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.195678 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.195657    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 21:12:38.196194 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.196165    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 21:12:38.196294 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.196216    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nvbrg\""
Sep 15 21:12:38.212320 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.212299    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"]
Sep 15 21:12:38.240296 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.240275    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"]
Sep 15 21:12:38.244337 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.244318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.246903 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.246885    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 21:12:38.247010 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.246968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nnjk2\""
Sep 15 21:12:38.247172 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.247154    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 21:12:38.247520 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.247505    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 21:12:38.252161 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.252141    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dj85v"]
Sep 15 21:12:38.256095 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.256078    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.260484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.260464    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 21:12:38.260565 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.260538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 21:12:38.260967 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.260943    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s6wpq\""
Sep 15 21:12:38.261056 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.260966    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 21:12:38.262237 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.262218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"]
Sep 15 21:12:38.327279 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/412e6858-6a69-46ca-8bf0-4002bf13f748-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.327279 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.327531 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61172647-ba80-42bd-9ef8-3a57a9afd08e-root\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.327531 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.327531 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-tls\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.327531 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.327531 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-accelerators-collector-config\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.327531 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfggk\" (UniqueName: \"kubernetes.io/projected/61172647-ba80-42bd-9ef8-3a57a9afd08e-kube-api-access-wfggk\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.327736 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61172647-ba80-42bd-9ef8-3a57a9afd08e-sys\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.327736 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/720e2ec7-2e6e-4807-9fe7-111093f9649a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.327736 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwxbb\" (UniqueName: \"kubernetes.io/projected/412e6858-6a69-46ca-8bf0-4002bf13f748-kube-api-access-qwxbb\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.327736 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/412e6858-6a69-46ca-8bf0-4002bf13f748-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.327736 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/720e2ec7-2e6e-4807-9fe7-111093f9649a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.327736 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7xbd\" (UniqueName: \"kubernetes.io/projected/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-api-access-b7xbd\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.327736 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/412e6858-6a69-46ca-8bf0-4002bf13f748-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.327955 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-wtmp\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.327955 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-textfile\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.327955 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61172647-ba80-42bd-9ef8-3a57a9afd08e-metrics-client-ca\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.327955 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.327801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.428876 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.428848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-wtmp\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.428876 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.428880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-textfile\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.429107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.428903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61172647-ba80-42bd-9ef8-3a57a9afd08e-metrics-client-ca\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.429107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.428927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.429107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.428967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/412e6858-6a69-46ca-8bf0-4002bf13f748-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.429107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.428995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.429107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61172647-ba80-42bd-9ef8-3a57a9afd08e-root\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.429107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-wtmp\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.429107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.429107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-tls\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.429107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.429107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-accelerators-collector-config\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfggk\" (UniqueName: \"kubernetes.io/projected/61172647-ba80-42bd-9ef8-3a57a9afd08e-kube-api-access-wfggk\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61172647-ba80-42bd-9ef8-3a57a9afd08e-sys\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/720e2ec7-2e6e-4807-9fe7-111093f9649a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:12:38.429199    2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwxbb\" (UniqueName: \"kubernetes.io/projected/412e6858-6a69-46ca-8bf0-4002bf13f748-kube-api-access-qwxbb\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/412e6858-6a69-46ca-8bf0-4002bf13f748-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-textfile\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/720e2ec7-2e6e-4807-9fe7-111093f9649a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:12:38.429316    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-tls podName:720e2ec7-2e6e-4807-9fe7-111093f9649a nodeName:}" failed. No retries permitted until 2026-09-15 21:12:38.929296544 +0000 UTC m=+148.892353907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-z5ft4" (UID: "720e2ec7-2e6e-4807-9fe7-111093f9649a") : secret "kube-state-metrics-tls" not found
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7xbd\" (UniqueName: \"kubernetes.io/projected/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-api-access-b7xbd\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.429625 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/412e6858-6a69-46ca-8bf0-4002bf13f748-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.430169 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61172647-ba80-42bd-9ef8-3a57a9afd08e-metrics-client-ca\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.430169 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-accelerators-collector-config\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.430169 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/720e2ec7-2e6e-4807-9fe7-111093f9649a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.430169 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.430169 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61172647-ba80-42bd-9ef8-3a57a9afd08e-sys\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.430169 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.429928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61172647-ba80-42bd-9ef8-3a57a9afd08e-root\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.430508 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.430486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/412e6858-6a69-46ca-8bf0-4002bf13f748-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.430699 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.430677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/720e2ec7-2e6e-4807-9fe7-111093f9649a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.431912 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.431875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/412e6858-6a69-46ca-8bf0-4002bf13f748-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.432020 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.431880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-tls\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.432376 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.432333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61172647-ba80-42bd-9ef8-3a57a9afd08e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.432376 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.432371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/412e6858-6a69-46ca-8bf0-4002bf13f748-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.432468 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.432333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.440528 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.440471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfggk\" (UniqueName: \"kubernetes.io/projected/61172647-ba80-42bd-9ef8-3a57a9afd08e-kube-api-access-wfggk\") pod \"node-exporter-dj85v\" (UID: \"61172647-ba80-42bd-9ef8-3a57a9afd08e\") " pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.440871 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.440854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwxbb\" (UniqueName: \"kubernetes.io/projected/412e6858-6a69-46ca-8bf0-4002bf13f748-kube-api-access-qwxbb\") pod \"openshift-state-metrics-67dcf8c979-lmht9\" (UID: \"412e6858-6a69-46ca-8bf0-4002bf13f748\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.441504 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.441488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7xbd\" (UniqueName: \"kubernetes.io/projected/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-api-access-b7xbd\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.501510 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.501490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"
Sep 15 21:12:38.564825 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.564799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dj85v"
Sep 15 21:12:38.577116 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:38.577093    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61172647_ba80_42bd_9ef8_3a57a9afd08e.slice/crio-270764fb60f52934b449a511e70c4ed6662aeb856da7edd1ccdd6c1c015d55e0 WatchSource:0}: Error finding container 270764fb60f52934b449a511e70c4ed6662aeb856da7edd1ccdd6c1c015d55e0: Status 404 returned error can't find the container with id 270764fb60f52934b449a511e70c4ed6662aeb856da7edd1ccdd6c1c015d55e0
Sep 15 21:12:38.773732 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.773708    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9"]
Sep 15 21:12:38.775804 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:38.775778    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412e6858_6a69_46ca_8bf0_4002bf13f748.slice/crio-17866a0e25a1487ce528f7a89221318f170f979cee0d04df0f2499a202e2a4d2 WatchSource:0}: Error finding container 17866a0e25a1487ce528f7a89221318f170f979cee0d04df0f2499a202e2a4d2: Status 404 returned error can't find the container with id 17866a0e25a1487ce528f7a89221318f170f979cee0d04df0f2499a202e2a4d2
Sep 15 21:12:38.933792 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.933768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:38.935905 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:38.935886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/720e2ec7-2e6e-4807-9fe7-111093f9649a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-z5ft4\" (UID: \"720e2ec7-2e6e-4807-9fe7-111093f9649a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:39.031646 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.031577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9" event={"ID":"412e6858-6a69-46ca-8bf0-4002bf13f748","Type":"ContainerStarted","Data":"500580e785f1f67f1958cb5a637a19f3d6b99945f73c117ae798ce9b57af0a8f"}
Sep 15 21:12:39.031745 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.031668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9" event={"ID":"412e6858-6a69-46ca-8bf0-4002bf13f748","Type":"ContainerStarted","Data":"26aede03d9c538dbe6cf18e321edd44e8afb35dd78470f42f073fe748e91e381"}
Sep 15 21:12:39.031745 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.031690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9" event={"ID":"412e6858-6a69-46ca-8bf0-4002bf13f748","Type":"ContainerStarted","Data":"17866a0e25a1487ce528f7a89221318f170f979cee0d04df0f2499a202e2a4d2"}
Sep 15 21:12:39.032601 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.032581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj85v" event={"ID":"61172647-ba80-42bd-9ef8-3a57a9afd08e","Type":"ContainerStarted","Data":"270764fb60f52934b449a511e70c4ed6662aeb856da7edd1ccdd6c1c015d55e0"}
Sep 15 21:12:39.152709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.152672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"
Sep 15 21:12:39.266192 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.266160    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:12:39.270461 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.270439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.273247 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.273223    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 21:12:39.273379 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.273224    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 21:12:39.273973 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.273953    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 21:12:39.274209 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.274186    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 21:12:39.275707 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.275138    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 21:12:39.275707 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.275152    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 21:12:39.275707 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.275211    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lp4nr\""
Sep 15 21:12:39.275707 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.275421    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 21:12:39.275707 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.275510    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 21:12:39.275707 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.275644    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 21:12:39.295085 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.294995    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:12:39.336761 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.336727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.336936 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.336782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.336936 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.336809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.336936 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.336839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-web-config\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.336936 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.336889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.336936 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.336935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-config-volume\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.337186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.336962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.337186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.336994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.337186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.337047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-config-out\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.337186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.337065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlh9\" (UniqueName: \"kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-kube-api-access-qmlh9\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.337186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.337099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.337186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.337127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.337186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.337143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.437173 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.437144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4"]
Sep 15 21:12:39.438011 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.437524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438011 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.437551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438011 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.437597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438011 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.437627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438011 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:12:39.437683    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-trusted-ca-bundle podName:91301a32-1458-4112-b53e-5d5aa1850078 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:39.937662619 +0000 UTC m=+149.900719998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "91301a32-1458-4112-b53e-5d5aa1850078") : configmap references non-existent config key: ca-bundle.crt
Sep 15 21:12:39.438382 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.438201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438382 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.438365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438495 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.438396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438495 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.438444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-web-config\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438495 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.438474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.438526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-config-volume\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.438550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.438591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.438620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-config-out\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.438864 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.438644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlh9\" (UniqueName: \"kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-kube-api-access-qmlh9\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.442123 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.442079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.445137 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.445084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.445231 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.445181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-web-config\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.445892 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.445689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.445892 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.445850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.446808 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.446429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.446808 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.446770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-config-volume\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.447149 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.447106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-config-out\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.447625 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.447561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.449882 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.449862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.455158 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.455139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlh9\" (UniqueName: \"kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-kube-api-access-qmlh9\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.943510 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.943465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:39.944210 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:39.944191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:40.041339 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:40.041301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj85v" event={"ID":"61172647-ba80-42bd-9ef8-3a57a9afd08e","Type":"ContainerStarted","Data":"5ccfd1969f03af1b7c36b80d8051bf47b46c5e92ac680333c101435b0be7e083"}
Sep 15 21:12:40.042894 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:40.042807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4" event={"ID":"720e2ec7-2e6e-4807-9fe7-111093f9649a","Type":"ContainerStarted","Data":"61340cda4a480647a6dfcf80806275f4a3f023e15797967de7238c7313a47f85"}
Sep 15 21:12:40.046840 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:40.046710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9" event={"ID":"412e6858-6a69-46ca-8bf0-4002bf13f748","Type":"ContainerStarted","Data":"cba8bf79d4726f06a0fa3dd2b236e761d8763e35e6ff4c067e029440ac79a292"}
Sep 15 21:12:40.080772 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:40.080712    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lmht9" podStartSLOduration=1.031665746 podStartE2EDuration="2.080690324s" totalImagesPullingTime="1.049024578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:40.080390805 +0000 UTC m=+150.043448190" watchObservedRunningTime="2026-09-15 21:12:40.080690324 +0000 UTC m=+150.043747706"
Sep 15 21:12:40.182627 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:40.182590    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:12:40.472046 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:40.472018    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:12:40.644197 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:40.644151    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91301a32_1458_4112_b53e_5d5aa1850078.slice/crio-d423e510d74fb315936c397864335e28e9f72257ee3cb3899a8acdf8187ab543 WatchSource:0}: Error finding container d423e510d74fb315936c397864335e28e9f72257ee3cb3899a8acdf8187ab543: Status 404 returned error can't find the container with id d423e510d74fb315936c397864335e28e9f72257ee3cb3899a8acdf8187ab543
Sep 15 21:12:41.052287 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:41.052202    2563 generic.go:356] "Generic (PLEG): container finished" podID="61172647-ba80-42bd-9ef8-3a57a9afd08e" containerID="5ccfd1969f03af1b7c36b80d8051bf47b46c5e92ac680333c101435b0be7e083" exitCode=0
Sep 15 21:12:41.052287 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:41.052276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj85v" event={"ID":"61172647-ba80-42bd-9ef8-3a57a9afd08e","Type":"ContainerDied","Data":"5ccfd1969f03af1b7c36b80d8051bf47b46c5e92ac680333c101435b0be7e083"}
Sep 15 21:12:41.054572 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:41.054500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4" event={"ID":"720e2ec7-2e6e-4807-9fe7-111093f9649a","Type":"ContainerStarted","Data":"ae14a9286fbf2499d70b738d767cd6c60480b7738599107e51268bfb3508d856"}
Sep 15 21:12:41.054681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:41.054604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4" event={"ID":"720e2ec7-2e6e-4807-9fe7-111093f9649a","Type":"ContainerStarted","Data":"5d3665b2e597c303fd4051b35b56e59ac6465a8fcb2e0ebcf800dcbb2bab3608"}
Sep 15 21:12:41.054681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:41.054632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4" event={"ID":"720e2ec7-2e6e-4807-9fe7-111093f9649a","Type":"ContainerStarted","Data":"644730fa787db38300a1d588b1b57baa929b0d9490baec55387d92067558772d"}
Sep 15 21:12:41.055780 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:41.055761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerStarted","Data":"d423e510d74fb315936c397864335e28e9f72257ee3cb3899a8acdf8187ab543"}
Sep 15 21:12:41.094364 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:41.094298    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z5ft4" podStartSLOduration=1.848906721 podStartE2EDuration="3.094276303s" totalImagesPullingTime="1.245369582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:41.092493867 +0000 UTC m=+151.055551250" watchObservedRunningTime="2026-09-15 21:12:41.094276303 +0000 UTC m=+151.057333684"
Sep 15 21:12:42.062187 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:42.062089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj85v" event={"ID":"61172647-ba80-42bd-9ef8-3a57a9afd08e","Type":"ContainerStarted","Data":"36ef642597524208cac1b8fae1a83e377344e037305d51b66a47b20fde3dea62"}
Sep 15 21:12:42.062187 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:42.062142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj85v" event={"ID":"61172647-ba80-42bd-9ef8-3a57a9afd08e","Type":"ContainerStarted","Data":"6a85339677e706a2ce64b109d0cab88f23ae58442a92c4444f60e86ce28e342f"}
Sep 15 21:12:42.063603 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:42.063577    2563 generic.go:356] "Generic (PLEG): container finished" podID="91301a32-1458-4112-b53e-5d5aa1850078" containerID="57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f" exitCode=0
Sep 15 21:12:42.063715 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:42.063660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerDied","Data":"57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f"}
Sep 15 21:12:42.084000 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:42.083372    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dj85v" podStartSLOduration=2.648476639 podStartE2EDuration="4.083340268s" totalImagesPullingTime="1.358726384s" totalInitContainerRuntime="76.137245ms" isStatefulPod=true podCreationTimestamp="2026-09-15 21:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:42.082988935 +0000 UTC m=+152.046046316" watchObservedRunningTime="2026-09-15 21:12:42.083340268 +0000 UTC m=+152.046397650"
Sep 15 21:12:43.425331 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.425293    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"]
Sep 15 21:12:43.432310 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.432283    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.434599 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.434574    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 15 21:12:43.434719 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.434685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 15 21:12:43.434795 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.434722    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-x6tbc\""
Sep 15 21:12:43.434795 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.434752    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 15 21:12:43.434891 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.434825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 15 21:12:43.434891 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.434874    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 15 21:12:43.440298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.440273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"]
Sep 15 21:12:43.440436 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.440380    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 15 21:12:43.573071 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.573031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb18cd9-b40f-457c-aecc-ac66416402c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.573071 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.573070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cb18cd9-b40f-457c-aecc-ac66416402c8-metrics-client-ca\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.573281 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.573096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb18cd9-b40f-457c-aecc-ac66416402c8-serving-certs-ca-bundle\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.573281 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.573114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsvhj\" (UniqueName: \"kubernetes.io/projected/7cb18cd9-b40f-457c-aecc-ac66416402c8-kube-api-access-nsvhj\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.573281 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.573171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.573281 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.573219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-federate-client-tls\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.573281 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.573244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-telemeter-client-tls\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.573452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.573297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-secret-telemeter-client\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.674158 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.674118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-secret-telemeter-client\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.674308 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.674196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb18cd9-b40f-457c-aecc-ac66416402c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.674308 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.674221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cb18cd9-b40f-457c-aecc-ac66416402c8-metrics-client-ca\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.674308 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.674249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb18cd9-b40f-457c-aecc-ac66416402c8-serving-certs-ca-bundle\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.674308 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.674272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsvhj\" (UniqueName: \"kubernetes.io/projected/7cb18cd9-b40f-457c-aecc-ac66416402c8-kube-api-access-nsvhj\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.674553 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.674313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.674553 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.674377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-federate-client-tls\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.674553 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.674409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-telemeter-client-tls\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.675169 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.675146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb18cd9-b40f-457c-aecc-ac66416402c8-serving-certs-ca-bundle\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.675295 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.675207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cb18cd9-b40f-457c-aecc-ac66416402c8-metrics-client-ca\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.675401 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.675333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cb18cd9-b40f-457c-aecc-ac66416402c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.676729 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.676703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-secret-telemeter-client\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.676922 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.676900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.677011 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.676993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-federate-client-tls\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.677081 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.677067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7cb18cd9-b40f-457c-aecc-ac66416402c8-telemeter-client-tls\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.683696 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.683671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsvhj\" (UniqueName: \"kubernetes.io/projected/7cb18cd9-b40f-457c-aecc-ac66416402c8-kube-api-access-nsvhj\") pod \"telemeter-client-7db6bd5cb5-nkb4b\" (UID: \"7cb18cd9-b40f-457c-aecc-ac66416402c8\") " pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:43.745295 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:43.745256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"
Sep 15 21:12:44.027282 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:44.027204    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b"]
Sep 15 21:12:44.030090 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:44.030060    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb18cd9_b40f_457c_aecc_ac66416402c8.slice/crio-451792f371d9cf1e9161b91dd3f4e44bc2a5c9bb481f0c19c28ee84995c86e47 WatchSource:0}: Error finding container 451792f371d9cf1e9161b91dd3f4e44bc2a5c9bb481f0c19c28ee84995c86e47: Status 404 returned error can't find the container with id 451792f371d9cf1e9161b91dd3f4e44bc2a5c9bb481f0c19c28ee84995c86e47
Sep 15 21:12:44.073594 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:44.073562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b" event={"ID":"7cb18cd9-b40f-457c-aecc-ac66416402c8","Type":"ContainerStarted","Data":"451792f371d9cf1e9161b91dd3f4e44bc2a5c9bb481f0c19c28ee84995c86e47"}
Sep 15 21:12:45.080381 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:45.080287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerStarted","Data":"67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec"}
Sep 15 21:12:45.080381 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:45.080372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerStarted","Data":"350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559"}
Sep 15 21:12:45.080719 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:45.080399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerStarted","Data":"45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01"}
Sep 15 21:12:45.080719 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:45.080422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerStarted","Data":"7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d"}
Sep 15 21:12:45.080719 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:45.080443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerStarted","Data":"d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef"}
Sep 15 21:12:46.092028 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:46.091999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerStarted","Data":"5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8"}
Sep 15 21:12:46.094102 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:46.094080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b" event={"ID":"7cb18cd9-b40f-457c-aecc-ac66416402c8","Type":"ContainerStarted","Data":"117b24de031539718f91eea6a4773ee37199abee6ad61c92e6d60ad2154b233d"}
Sep 15 21:12:46.094205 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:46.094121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b" event={"ID":"7cb18cd9-b40f-457c-aecc-ac66416402c8","Type":"ContainerStarted","Data":"59e40f74c0c3772abea7f244e1ea6c48dd4e55f19deb62fb311b9a6075f127b5"}
Sep 15 21:12:46.094205 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:46.094142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b" event={"ID":"7cb18cd9-b40f-457c-aecc-ac66416402c8","Type":"ContainerStarted","Data":"3f927fd148051f2abcd5d3a1778df8113465357ae8820a900d8ce5f199c48e07"}
Sep 15 21:12:46.119850 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:46.119808    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.584113759 podStartE2EDuration="7.119792688s" totalImagesPullingTime="4.519818715s" totalInitContainerRuntime="15.860214ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:46.118609861 +0000 UTC m=+156.081667241" watchObservedRunningTime="2026-09-15 21:12:46.119792688 +0000 UTC m=+156.082850069"
Sep 15 21:12:46.140871 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:46.140824    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7db6bd5cb5-nkb4b" podStartSLOduration=1.298839782 podStartE2EDuration="3.140807825s" totalImagesPullingTime="1.841968043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:46.139984306 +0000 UTC m=+156.103041689" watchObservedRunningTime="2026-09-15 21:12:46.140807825 +0000 UTC m=+156.103865206"
Sep 15 21:12:47.365097 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:12:47.365050    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-m9jvn" podUID="04d67dbc-688d-42da-a42a-66aa0b7258e8"
Sep 15 21:12:48.101911 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:48.101882    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:12:51.694444 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:51.694409    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-lsb6g"]
Sep 15 21:12:51.698773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:51.698756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-lsb6g"
Sep 15 21:12:51.700824 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:51.700796    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 21:12:51.700961 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:51.700836    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 21:12:51.701008 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:51.700984    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-shvcg\""
Sep 15 21:12:51.708757 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:51.708735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-lsb6g"]
Sep 15 21:12:51.851373 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:51.851315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnsfq\" (UniqueName: \"kubernetes.io/projected/7fda35c6-f9a2-4c7b-89d7-74a1a74b6847-kube-api-access-fnsfq\") pod \"downloads-76f99584f4-lsb6g\" (UID: \"7fda35c6-f9a2-4c7b-89d7-74a1a74b6847\") " pod="openshift-console/downloads-76f99584f4-lsb6g"
Sep 15 21:12:51.952324 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:51.952250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnsfq\" (UniqueName: \"kubernetes.io/projected/7fda35c6-f9a2-4c7b-89d7-74a1a74b6847-kube-api-access-fnsfq\") pod \"downloads-76f99584f4-lsb6g\" (UID: \"7fda35c6-f9a2-4c7b-89d7-74a1a74b6847\") " pod="openshift-console/downloads-76f99584f4-lsb6g"
Sep 15 21:12:51.961554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:51.961528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnsfq\" (UniqueName: \"kubernetes.io/projected/7fda35c6-f9a2-4c7b-89d7-74a1a74b6847-kube-api-access-fnsfq\") pod \"downloads-76f99584f4-lsb6g\" (UID: \"7fda35c6-f9a2-4c7b-89d7-74a1a74b6847\") " pod="openshift-console/downloads-76f99584f4-lsb6g"
Sep 15 21:12:52.007588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:52.007564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-lsb6g"
Sep 15 21:12:52.254452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:52.254365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:12:52.256798 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:52.256772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04d67dbc-688d-42da-a42a-66aa0b7258e8-metrics-tls\") pod \"dns-default-m9jvn\" (UID: \"04d67dbc-688d-42da-a42a-66aa0b7258e8\") " pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:12:52.275634 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:52.275611    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-lsb6g"]
Sep 15 21:12:52.277988 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:52.277959    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fda35c6_f9a2_4c7b_89d7_74a1a74b6847.slice/crio-7ac3bd53c8357f5761bcf3bba9d2cf9e79beee964c891c77f8279a35e0f49f0a WatchSource:0}: Error finding container 7ac3bd53c8357f5761bcf3bba9d2cf9e79beee964c891c77f8279a35e0f49f0a: Status 404 returned error can't find the container with id 7ac3bd53c8357f5761bcf3bba9d2cf9e79beee964c891c77f8279a35e0f49f0a
Sep 15 21:12:52.305770 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:52.305750    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nfdpp\""
Sep 15 21:12:52.313952 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:52.313937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:12:52.581505 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:52.581379    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m9jvn"]
Sep 15 21:12:52.583974 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:52.583952    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d67dbc_688d_42da_a42a_66aa0b7258e8.slice/crio-66606da606092bb3818477d82730e64a70bfbde17cb649b5ec33ede444baba87 WatchSource:0}: Error finding container 66606da606092bb3818477d82730e64a70bfbde17cb649b5ec33ede444baba87: Status 404 returned error can't find the container with id 66606da606092bb3818477d82730e64a70bfbde17cb649b5ec33ede444baba87
Sep 15 21:12:53.121523 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:53.121491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m9jvn" event={"ID":"04d67dbc-688d-42da-a42a-66aa0b7258e8","Type":"ContainerStarted","Data":"66606da606092bb3818477d82730e64a70bfbde17cb649b5ec33ede444baba87"}
Sep 15 21:12:53.122790 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:53.122759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-lsb6g" event={"ID":"7fda35c6-f9a2-4c7b-89d7-74a1a74b6847","Type":"ContainerStarted","Data":"7ac3bd53c8357f5761bcf3bba9d2cf9e79beee964c891c77f8279a35e0f49f0a"}
Sep 15 21:12:55.133963 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:55.133928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m9jvn" event={"ID":"04d67dbc-688d-42da-a42a-66aa0b7258e8","Type":"ContainerStarted","Data":"4aa8afd0bd92f4b19e0a0f8f7a30867a320476fe0816dd6db5f717c6adb6156d"}
Sep 15 21:12:55.134400 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:55.133985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m9jvn" event={"ID":"04d67dbc-688d-42da-a42a-66aa0b7258e8","Type":"ContainerStarted","Data":"98cf014a6fc0291f66f6e1e319957db5912c1d42f010e27ae5d3f6b30fab0ac5"}
Sep 15 21:12:55.134400 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:55.134079    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:12:55.153040 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:55.152989    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m9jvn" podStartSLOduration=129.5942698 podStartE2EDuration="2m11.152967095s" totalImagesPullingTime="1.558697295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:55.150985792 +0000 UTC m=+165.114043173" watchObservedRunningTime="2026-09-15 21:12:55.152967095 +0000 UTC m=+165.116024476"
Sep 15 21:12:56.140849 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.140817    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd1f9f59-883a-4b1c-80af-41e5ebc914f0" containerID="a9e84a5f9180700faad9b0bb57f172320e2d7d0589e58bd08892164f5a167cad" exitCode=0
Sep 15 21:12:56.141298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.140899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wzf6l" event={"ID":"fd1f9f59-883a-4b1c-80af-41e5ebc914f0","Type":"ContainerDied","Data":"a9e84a5f9180700faad9b0bb57f172320e2d7d0589e58bd08892164f5a167cad"}
Sep 15 21:12:56.141553 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.141534    2563 scope.go:117] "RemoveContainer" containerID="a9e84a5f9180700faad9b0bb57f172320e2d7d0589e58bd08892164f5a167cad"
Sep 15 21:12:56.217577 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.217552    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d4cdcb9c6-hzsxj"]
Sep 15 21:12:56.229916 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.229386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.232783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.231956    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 21:12:56.232783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.232204    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 21:12:56.232783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.232301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 21:12:56.232783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.232509    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 21:12:56.232783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.232522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 21:12:56.232783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.232732    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4cdcb9c6-hzsxj"]
Sep 15 21:12:56.233157 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.232764    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fj6tq\""
Sep 15 21:12:56.389883 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.389847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-oauth-config\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.390065 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.389895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt642\" (UniqueName: \"kubernetes.io/projected/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-kube-api-access-xt642\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.390065 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.389920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-serving-cert\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.390065 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.389954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-service-ca\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.390065 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.390062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-config\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.390272 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.390087    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-oauth-serving-cert\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.490996 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.490911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-config\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.490996 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.490956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-oauth-serving-cert\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.490996 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.490977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-oauth-config\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.491248 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.491003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt642\" (UniqueName: \"kubernetes.io/projected/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-kube-api-access-xt642\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.491248 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.491021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-serving-cert\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.491248 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.491045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-service-ca\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.491916 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.491876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-oauth-serving-cert\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.492049 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.491920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-config\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.492049 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.491996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-service-ca\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.494097 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.494052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-oauth-config\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.494427 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.494405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-serving-cert\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.502249 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.502225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt642\" (UniqueName: \"kubernetes.io/projected/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-kube-api-access-xt642\") pod \"console-7d4cdcb9c6-hzsxj\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") " pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.545050 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.545012    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:12:56.832184 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:56.832095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4cdcb9c6-hzsxj"]
Sep 15 21:12:56.838722 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:12:56.838532    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109129cc_3e02_4bb2_85b0_9fe6bbbb10fa.slice/crio-7d9a9449e5609700ca7d5c75e06ff4c82b2efb3f80beadf2a0842c030242a6a6 WatchSource:0}: Error finding container 7d9a9449e5609700ca7d5c75e06ff4c82b2efb3f80beadf2a0842c030242a6a6: Status 404 returned error can't find the container with id 7d9a9449e5609700ca7d5c75e06ff4c82b2efb3f80beadf2a0842c030242a6a6
Sep 15 21:12:57.147220 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:57.147181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4cdcb9c6-hzsxj" event={"ID":"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa","Type":"ContainerStarted","Data":"7d9a9449e5609700ca7d5c75e06ff4c82b2efb3f80beadf2a0842c030242a6a6"}
Sep 15 21:12:57.149588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:12:57.149556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wzf6l" event={"ID":"fd1f9f59-883a-4b1c-80af-41e5ebc914f0","Type":"ContainerStarted","Data":"f2de6fffed54e61eac0146301dc569bdb4fd7d4cfbc5af3f714d38f79d7f2635"}
Sep 15 21:13:00.166311 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:00.166269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4cdcb9c6-hzsxj" event={"ID":"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa","Type":"ContainerStarted","Data":"b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4"}
Sep 15 21:13:00.183982 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:00.183921    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d4cdcb9c6-hzsxj" podStartSLOduration=1.07257621 podStartE2EDuration="4.183901649s" totalImagesPullingTime="3.111325439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:00.182800688 +0000 UTC m=+170.145858071" watchObservedRunningTime="2026-09-15 21:13:00.183901649 +0000 UTC m=+170.146959031"
Sep 15 21:13:04.598621 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.598533    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c59bb7d79-t6skv"]
Sep 15 21:13:04.606367 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.606326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.613536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.613494    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c59bb7d79-t6skv"]
Sep 15 21:13:04.614374 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.614328    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 21:13:04.671083 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.671047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-oauth-config\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.671262 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.671096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7gr\" (UniqueName: \"kubernetes.io/projected/dc65c4e6-ee38-438c-a23b-5150746d54ef-kube-api-access-bj7gr\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.671262 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.671145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-service-ca\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.671262 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.671170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-config\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.671262 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.671186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-trusted-ca-bundle\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.671262 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.671207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-oauth-serving-cert\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.671523 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.671288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-serving-cert\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.771989 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.771948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-oauth-config\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.772159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.771999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7gr\" (UniqueName: \"kubernetes.io/projected/dc65c4e6-ee38-438c-a23b-5150746d54ef-kube-api-access-bj7gr\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.772159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.772045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-service-ca\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.772159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.772072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-config\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.772159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.772094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-trusted-ca-bundle\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.772159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.772117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-oauth-serving-cert\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.772159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.772154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-serving-cert\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.773013 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.772940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-service-ca\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.773013 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.772967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-config\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.773231 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.773206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-trusted-ca-bundle\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.773297 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.773212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-oauth-serving-cert\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.775104 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.775082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-serving-cert\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.775203 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.775156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-oauth-config\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.781211 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.781191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7gr\" (UniqueName: \"kubernetes.io/projected/dc65c4e6-ee38-438c-a23b-5150746d54ef-kube-api-access-bj7gr\") pod \"console-5c59bb7d79-t6skv\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") " pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:04.922987 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:04.922951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:05.143507 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:05.143471    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m9jvn"
Sep 15 21:13:06.545883 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:06.545850    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:13:06.546284 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:06.546133    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:13:06.550982 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:06.550959    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:13:07.205487 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:07.205453    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:13:09.211626 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:09.211595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-lsb6g" event={"ID":"7fda35c6-f9a2-4c7b-89d7-74a1a74b6847","Type":"ContainerStarted","Data":"4d61caf117700691c006e1ef54bfcf42abb16e86d075345a685db4a1bf7698dc"}
Sep 15 21:13:09.229657 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:09.229599    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-lsb6g" podStartSLOduration=1.372848985 podStartE2EDuration="18.229584578s" totalImagesPullingTime="16.856735593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:09.228035585 +0000 UTC m=+179.191092989" watchObservedRunningTime="2026-09-15 21:13:09.229584578 +0000 UTC m=+179.192641957"
Sep 15 21:13:09.311933 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:09.311824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c59bb7d79-t6skv"]
Sep 15 21:13:09.315046 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:13:09.315009    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc65c4e6_ee38_438c_a23b_5150746d54ef.slice/crio-79464c3904bab81b835e62fde01afa7bd40d95c07136264cc82fb1f264b33983 WatchSource:0}: Error finding container 79464c3904bab81b835e62fde01afa7bd40d95c07136264cc82fb1f264b33983: Status 404 returned error can't find the container with id 79464c3904bab81b835e62fde01afa7bd40d95c07136264cc82fb1f264b33983
Sep 15 21:13:10.224035 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:10.223998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c59bb7d79-t6skv" event={"ID":"dc65c4e6-ee38-438c-a23b-5150746d54ef","Type":"ContainerStarted","Data":"ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542"}
Sep 15 21:13:10.224635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:10.224601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c59bb7d79-t6skv" event={"ID":"dc65c4e6-ee38-438c-a23b-5150746d54ef","Type":"ContainerStarted","Data":"79464c3904bab81b835e62fde01afa7bd40d95c07136264cc82fb1f264b33983"}
Sep 15 21:13:10.224760 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:10.224647    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-lsb6g"
Sep 15 21:13:10.243154 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:10.243096    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c59bb7d79-t6skv" podStartSLOduration=6.243076 podStartE2EDuration="6.243076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:10.240840379 +0000 UTC m=+180.203897760" watchObservedRunningTime="2026-09-15 21:13:10.243076 +0000 UTC m=+180.206133384"
Sep 15 21:13:10.245416 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:10.245387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-lsb6g"
Sep 15 21:13:14.923089 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:14.923053    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:14.923665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:14.923127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:14.928388 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:14.928361    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:15.251938 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:15.251854    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:13:15.297688 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:15.297648    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4cdcb9c6-hzsxj"]
Sep 15 21:13:26.294289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:26.294209    2563 generic.go:356] "Generic (PLEG): container finished" podID="37eb94cb-0bca-47f2-bcc0-7bcda93efd65" containerID="55118f112c1551b105b331704552d178231157167325be9793a2b81600cc3888" exitCode=0
Sep 15 21:13:26.294683 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:26.294280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj" event={"ID":"37eb94cb-0bca-47f2-bcc0-7bcda93efd65","Type":"ContainerDied","Data":"55118f112c1551b105b331704552d178231157167325be9793a2b81600cc3888"}
Sep 15 21:13:26.294683 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:26.294616    2563 scope.go:117] "RemoveContainer" containerID="55118f112c1551b105b331704552d178231157167325be9793a2b81600cc3888"
Sep 15 21:13:27.305405 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:27.305371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qzndj" event={"ID":"37eb94cb-0bca-47f2-bcc0-7bcda93efd65","Type":"ContainerStarted","Data":"d07bfe8e91998f16bc55b8371c1853068ec4949b1c7a5067ba66710640370fa1"}
Sep 15 21:13:30.980729 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:13:30.980688    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7474e7_64cf_464e_b7f4_d8114506b756.slice/crio-621dfac6363467bad6086fe6989bee31943fc98ff9f1c20d8d4b85bba30339ca.scope\": RecentStats: unable to find data in memory cache]"
Sep 15 21:13:31.323093 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:31.323006    2563 generic.go:356] "Generic (PLEG): container finished" podID="5e7474e7-64cf-464e-b7f4-d8114506b756" containerID="621dfac6363467bad6086fe6989bee31943fc98ff9f1c20d8d4b85bba30339ca" exitCode=0
Sep 15 21:13:31.323235 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:31.323087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7" event={"ID":"5e7474e7-64cf-464e-b7f4-d8114506b756","Type":"ContainerDied","Data":"621dfac6363467bad6086fe6989bee31943fc98ff9f1c20d8d4b85bba30339ca"}
Sep 15 21:13:31.323505 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:31.323489    2563 scope.go:117] "RemoveContainer" containerID="621dfac6363467bad6086fe6989bee31943fc98ff9f1c20d8d4b85bba30339ca"
Sep 15 21:13:32.329386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:32.329337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-q6lc7" event={"ID":"5e7474e7-64cf-464e-b7f4-d8114506b756","Type":"ContainerStarted","Data":"413f2036a7b7f7f75e55b2b02adc48523700b386397014f10a4db8a8ce3fd9e3"}
Sep 15 21:13:40.320584 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.320541    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d4cdcb9c6-hzsxj" podUID="109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" containerName="console" containerID="cri-o://b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4" gracePeriod=15
Sep 15 21:13:40.585369 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.585305    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:13:40.698472 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.698447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt642\" (UniqueName: \"kubernetes.io/projected/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-kube-api-access-xt642\") pod \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") "
Sep 15 21:13:40.698472 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.698481    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-serving-cert\") pod \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") "
Sep 15 21:13:40.698669 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.698512    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-oauth-serving-cert\") pod \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") "
Sep 15 21:13:40.698669 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.698543    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-config\") pod \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") "
Sep 15 21:13:40.698669 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.698563    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-oauth-config\") pod \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") "
Sep 15 21:13:40.698669 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.698580    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-service-ca\") pod \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\" (UID: \"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa\") "
Sep 15 21:13:40.698999 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.698969    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-config" (OuterVolumeSpecName: "console-config") pod "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" (UID: "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:13:40.698999 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.698978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-service-ca" (OuterVolumeSpecName: "service-ca") pod "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" (UID: "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:13:40.699155 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.698966    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" (UID: "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:13:40.700766 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.700736    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" (UID: "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:40.700766 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.700748    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-kube-api-access-xt642" (OuterVolumeSpecName: "kube-api-access-xt642") pod "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" (UID: "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa"). InnerVolumeSpecName "kube-api-access-xt642". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:13:40.700892 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.700774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" (UID: "109129cc-3e02-4bb2-85b0-9fe6bbbb10fa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:40.799673 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.799652    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt642\" (UniqueName: \"kubernetes.io/projected/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-kube-api-access-xt642\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:40.799673 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.799672    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-serving-cert\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:40.799783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.799681    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-oauth-serving-cert\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:40.799783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.799689    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-config\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:40.799783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.799697    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-console-oauth-config\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:40.799783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:40.799705    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa-service-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:41.363900 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:41.363871    2563 generic.go:356] "Generic (PLEG): container finished" podID="109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" containerID="b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4" exitCode=0
Sep 15 21:13:41.364273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:41.363916    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4cdcb9c6-hzsxj"
Sep 15 21:13:41.364273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:41.363958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4cdcb9c6-hzsxj" event={"ID":"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa","Type":"ContainerDied","Data":"b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4"}
Sep 15 21:13:41.364273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:41.364007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4cdcb9c6-hzsxj" event={"ID":"109129cc-3e02-4bb2-85b0-9fe6bbbb10fa","Type":"ContainerDied","Data":"7d9a9449e5609700ca7d5c75e06ff4c82b2efb3f80beadf2a0842c030242a6a6"}
Sep 15 21:13:41.364273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:41.364028    2563 scope.go:117] "RemoveContainer" containerID="b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4"
Sep 15 21:13:41.372070 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:41.372055    2563 scope.go:117] "RemoveContainer" containerID="b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4"
Sep 15 21:13:41.372354 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:13:41.372323    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4\": container with ID starting with b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4 not found: ID does not exist" containerID="b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4"
Sep 15 21:13:41.372422 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:41.372365    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4"} err="failed to get container status \"b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4\": rpc error: code = NotFound desc = could not find container \"b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4\": container with ID starting with b978bfd3475a739355bdb4bbb366de954dcf822e645195bd2adb4f3f4d8798d4 not found: ID does not exist"
Sep 15 21:13:41.384014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:41.383994    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4cdcb9c6-hzsxj"]
Sep 15 21:13:41.388112 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:41.388094    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d4cdcb9c6-hzsxj"]
Sep 15 21:13:42.524363 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:42.524319    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" path="/var/lib/kubelet/pods/109129cc-3e02-4bb2-85b0-9fe6bbbb10fa/volumes"
Sep 15 21:13:58.529524 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:58.529487    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:13:58.530023 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:58.529794    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="alertmanager" containerID="cri-o://d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef" gracePeriod=120
Sep 15 21:13:58.530023 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:58.529883    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="config-reloader" containerID="cri-o://7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d" gracePeriod=120
Sep 15 21:13:58.530023 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:58.529901    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy" containerID="cri-o://350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559" gracePeriod=120
Sep 15 21:13:58.530023 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:58.529964    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="prom-label-proxy" containerID="cri-o://5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8" gracePeriod=120
Sep 15 21:13:58.530253 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:58.529872    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy-web" containerID="cri-o://45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01" gracePeriod=120
Sep 15 21:13:58.530253 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:58.529871    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy-metric" containerID="cri-o://67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec" gracePeriod=120
Sep 15 21:13:59.437981 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.437951    2563 generic.go:356] "Generic (PLEG): container finished" podID="91301a32-1458-4112-b53e-5d5aa1850078" containerID="5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8" exitCode=0
Sep 15 21:13:59.437981 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.437973    2563 generic.go:356] "Generic (PLEG): container finished" podID="91301a32-1458-4112-b53e-5d5aa1850078" containerID="350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559" exitCode=0
Sep 15 21:13:59.437981 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.437979    2563 generic.go:356] "Generic (PLEG): container finished" podID="91301a32-1458-4112-b53e-5d5aa1850078" containerID="7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d" exitCode=0
Sep 15 21:13:59.437981 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.437986    2563 generic.go:356] "Generic (PLEG): container finished" podID="91301a32-1458-4112-b53e-5d5aa1850078" containerID="d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef" exitCode=0
Sep 15 21:13:59.438195 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.438006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerDied","Data":"5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8"}
Sep 15 21:13:59.438195 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.438055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerDied","Data":"350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559"}
Sep 15 21:13:59.438195 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.438082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerDied","Data":"7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d"}
Sep 15 21:13:59.438195 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.438105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerDied","Data":"d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef"}
Sep 15 21:13:59.763999 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.763978    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:13:59.822792 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.822761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.822937 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.822812    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-web-config\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.822937 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.822856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-trusted-ca-bundle\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.822937 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.822888    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-config-out\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.822937 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.822919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-web\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.823129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.822944    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-cluster-tls-config\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.823129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.822982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-main-tls\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.823129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.823034    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-tls-assets\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.823129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.823068    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-config-volume\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.823129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.823101    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmlh9\" (UniqueName: \"kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-kube-api-access-qmlh9\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.823129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.823128    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-main-db\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.823448 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.823172    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-metric\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.823448 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.823229    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-metrics-client-ca\") pod \"91301a32-1458-4112-b53e-5d5aa1850078\" (UID: \"91301a32-1458-4112-b53e-5d5aa1850078\") "
Sep 15 21:13:59.823448 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.823379    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:13:59.823611 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.823530    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.824332 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.824285    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:13:59.824651 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.824629    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:13:59.825611 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.825578    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:59.825849 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.825799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-config-out" (OuterVolumeSpecName: "config-out") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:13:59.826421 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.826394    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:13:59.826648 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.826622    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:59.827208 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.827145    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:59.827938 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.827864    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:59.828036 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.828015    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-config-volume" (OuterVolumeSpecName: "config-volume") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:59.828466 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.828433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-kube-api-access-qmlh9" (OuterVolumeSpecName: "kube-api-access-qmlh9") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "kube-api-access-qmlh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:13:59.830719 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.830692    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:59.837188 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.837166    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-web-config" (OuterVolumeSpecName: "web-config") pod "91301a32-1458-4112-b53e-5d5aa1850078" (UID: "91301a32-1458-4112-b53e-5d5aa1850078"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:13:59.924463 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924442    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924463 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924461    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-web-config\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924472    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-config-out\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924480    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924488    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-cluster-tls-config\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924496    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-main-tls\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924504    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-tls-assets\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924511    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-config-volume\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924519    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlh9\" (UniqueName: \"kubernetes.io/projected/91301a32-1458-4112-b53e-5d5aa1850078-kube-api-access-qmlh9\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924526    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91301a32-1458-4112-b53e-5d5aa1850078-alertmanager-main-db\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924537    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91301a32-1458-4112-b53e-5d5aa1850078-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:13:59.924568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:13:59.924545    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91301a32-1458-4112-b53e-5d5aa1850078-metrics-client-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:14:00.444757 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.444729    2563 generic.go:356] "Generic (PLEG): container finished" podID="91301a32-1458-4112-b53e-5d5aa1850078" containerID="67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec" exitCode=0
Sep 15 21:14:00.444757 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.444751    2563 generic.go:356] "Generic (PLEG): container finished" podID="91301a32-1458-4112-b53e-5d5aa1850078" containerID="45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01" exitCode=0
Sep 15 21:14:00.444919 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.444809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.444919 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.444809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerDied","Data":"67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec"}
Sep 15 21:14:00.444919 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.444874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerDied","Data":"45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01"}
Sep 15 21:14:00.444919 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.444904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91301a32-1458-4112-b53e-5d5aa1850078","Type":"ContainerDied","Data":"d423e510d74fb315936c397864335e28e9f72257ee3cb3899a8acdf8187ab543"}
Sep 15 21:14:00.445038 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.444934    2563 scope.go:117] "RemoveContainer" containerID="5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8"
Sep 15 21:14:00.453320 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.453304    2563 scope.go:117] "RemoveContainer" containerID="67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec"
Sep 15 21:14:00.460140 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.460126    2563 scope.go:117] "RemoveContainer" containerID="350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559"
Sep 15 21:14:00.467087 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.467051    2563 scope.go:117] "RemoveContainer" containerID="45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01"
Sep 15 21:14:00.468714 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.468696    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:14:00.474076 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.474056    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:14:00.474134 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.474096    2563 scope.go:117] "RemoveContainer" containerID="7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d"
Sep 15 21:14:00.480604 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.480590    2563 scope.go:117] "RemoveContainer" containerID="d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef"
Sep 15 21:14:00.487023 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.487009    2563 scope.go:117] "RemoveContainer" containerID="57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f"
Sep 15 21:14:00.493529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.493513    2563 scope.go:117] "RemoveContainer" containerID="5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8"
Sep 15 21:14:00.493763 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:14:00.493746    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8\": container with ID starting with 5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8 not found: ID does not exist" containerID="5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8"
Sep 15 21:14:00.493831 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.493768    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8"} err="failed to get container status \"5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8\": rpc error: code = NotFound desc = could not find container \"5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8\": container with ID starting with 5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8 not found: ID does not exist"
Sep 15 21:14:00.493831 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.493784    2563 scope.go:117] "RemoveContainer" containerID="67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec"
Sep 15 21:14:00.494025 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:14:00.494006    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec\": container with ID starting with 67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec not found: ID does not exist" containerID="67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec"
Sep 15 21:14:00.494063 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.494032    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec"} err="failed to get container status \"67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec\": rpc error: code = NotFound desc = could not find container \"67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec\": container with ID starting with 67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec not found: ID does not exist"
Sep 15 21:14:00.494063 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.494053    2563 scope.go:117] "RemoveContainer" containerID="350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559"
Sep 15 21:14:00.494288 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:14:00.494270    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559\": container with ID starting with 350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559 not found: ID does not exist" containerID="350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559"
Sep 15 21:14:00.494357 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.494299    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559"} err="failed to get container status \"350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559\": rpc error: code = NotFound desc = could not find container \"350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559\": container with ID starting with 350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559 not found: ID does not exist"
Sep 15 21:14:00.494357 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.494313    2563 scope.go:117] "RemoveContainer" containerID="45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01"
Sep 15 21:14:00.494570 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:14:00.494556    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01\": container with ID starting with 45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01 not found: ID does not exist" containerID="45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01"
Sep 15 21:14:00.494609 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.494572    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01"} err="failed to get container status \"45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01\": rpc error: code = NotFound desc = could not find container \"45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01\": container with ID starting with 45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01 not found: ID does not exist"
Sep 15 21:14:00.494609 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.494583    2563 scope.go:117] "RemoveContainer" containerID="7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d"
Sep 15 21:14:00.494819 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:14:00.494803    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d\": container with ID starting with 7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d not found: ID does not exist" containerID="7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d"
Sep 15 21:14:00.494860 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.494824    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d"} err="failed to get container status \"7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d\": rpc error: code = NotFound desc = could not find container \"7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d\": container with ID starting with 7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d not found: ID does not exist"
Sep 15 21:14:00.494860 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.494836    2563 scope.go:117] "RemoveContainer" containerID="d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef"
Sep 15 21:14:00.495062 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:14:00.495040    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef\": container with ID starting with d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef not found: ID does not exist" containerID="d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef"
Sep 15 21:14:00.495151 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.495063    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef"} err="failed to get container status \"d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef\": rpc error: code = NotFound desc = could not find container \"d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef\": container with ID starting with d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef not found: ID does not exist"
Sep 15 21:14:00.495151 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.495074    2563 scope.go:117] "RemoveContainer" containerID="57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f"
Sep 15 21:14:00.495288 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:14:00.495266    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f\": container with ID starting with 57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f not found: ID does not exist" containerID="57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f"
Sep 15 21:14:00.495330 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.495291    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f"} err="failed to get container status \"57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f\": rpc error: code = NotFound desc = could not find container \"57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f\": container with ID starting with 57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f not found: ID does not exist"
Sep 15 21:14:00.495330 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.495303    2563 scope.go:117] "RemoveContainer" containerID="5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8"
Sep 15 21:14:00.495510 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.495495    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8"} err="failed to get container status \"5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8\": rpc error: code = NotFound desc = could not find container \"5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8\": container with ID starting with 5b0671ca5b957d3ab231a08e528e31e69e9fd9b42e4f3124a27e6ef7f8df0eb8 not found: ID does not exist"
Sep 15 21:14:00.495555 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.495510    2563 scope.go:117] "RemoveContainer" containerID="67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec"
Sep 15 21:14:00.495691 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.495676    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec"} err="failed to get container status \"67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec\": rpc error: code = NotFound desc = could not find container \"67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec\": container with ID starting with 67bad5ba28bac0e49fcb4b6399219a470b765212f79aa280a425f0cb58c25aec not found: ID does not exist"
Sep 15 21:14:00.495732 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.495690    2563 scope.go:117] "RemoveContainer" containerID="350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559"
Sep 15 21:14:00.495866 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.495852    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559"} err="failed to get container status \"350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559\": rpc error: code = NotFound desc = could not find container \"350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559\": container with ID starting with 350553e5c2d0325d39533351a5713aec62c468fbf9821c5d2d8678dcc3ff9559 not found: ID does not exist"
Sep 15 21:14:00.495913 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.495866    2563 scope.go:117] "RemoveContainer" containerID="45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01"
Sep 15 21:14:00.496055 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.496040    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01"} err="failed to get container status \"45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01\": rpc error: code = NotFound desc = could not find container \"45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01\": container with ID starting with 45a70f86ef020b564d49c24f3c02f935aa236a63ffb13f01f93b1c4f4d7bdf01 not found: ID does not exist"
Sep 15 21:14:00.496098 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.496056    2563 scope.go:117] "RemoveContainer" containerID="7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d"
Sep 15 21:14:00.496243 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.496229    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d"} err="failed to get container status \"7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d\": rpc error: code = NotFound desc = could not find container \"7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d\": container with ID starting with 7dc5c522614ec0366daa03acf30bc3d712ded7b0a9f44f7cac81fe1bc566d56d not found: ID does not exist"
Sep 15 21:14:00.496285 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.496243    2563 scope.go:117] "RemoveContainer" containerID="d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef"
Sep 15 21:14:00.496467 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.496450    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef"} err="failed to get container status \"d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef\": rpc error: code = NotFound desc = could not find container \"d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef\": container with ID starting with d31334e407853c015b5f263c3b395ef4e558902733c70520738fc9779cd592ef not found: ID does not exist"
Sep 15 21:14:00.496536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.496468    2563 scope.go:117] "RemoveContainer" containerID="57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f"
Sep 15 21:14:00.496655 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.496641    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f"} err="failed to get container status \"57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f\": rpc error: code = NotFound desc = could not find container \"57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f\": container with ID starting with 57c92de7169fdb5e1a063afb2f379c06fed5f764acc5126a567b5df8bc9edf9f not found: ID does not exist"
Sep 15 21:14:00.501671 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.501652    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:14:00.501966 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.501948    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="init-config-reloader"
Sep 15 21:14:00.502034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.501967    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="init-config-reloader"
Sep 15 21:14:00.502034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.501984    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy-metric"
Sep 15 21:14:00.502034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.501993    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy-metric"
Sep 15 21:14:00.502034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502006    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="alertmanager"
Sep 15 21:14:00.502034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502015    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="alertmanager"
Sep 15 21:14:00.502034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502025    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="config-reloader"
Sep 15 21:14:00.502034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502032    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="config-reloader"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502042    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy-web"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502050    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy-web"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502058    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502067    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502076    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" containerName="console"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502084    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" containerName="console"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502102    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="prom-label-proxy"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502112    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="prom-label-proxy"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502189    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="alertmanager"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502201    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy-web"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502210    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="prom-label-proxy"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502221    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy-metric"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502229    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="109129cc-3e02-4bb2-85b0-9fe6bbbb10fa" containerName="console"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502238    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="kube-rbac-proxy"
Sep 15 21:14:00.502406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.502246    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="91301a32-1458-4112-b53e-5d5aa1850078" containerName="config-reloader"
Sep 15 21:14:00.505683 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.505666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.507757 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.507729    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 21:14:00.507858 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.507794    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 21:14:00.507858 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.507810    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 21:14:00.508030 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.508014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 21:14:00.508131 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.508078    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 21:14:00.508189 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.508112    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 21:14:00.508189 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.508144    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lp4nr\""
Sep 15 21:14:00.508189 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.508155    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 21:14:00.508316 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.508115    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 21:14:00.517395 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.517371    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 21:14:00.525661 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.525638    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91301a32-1458-4112-b53e-5d5aa1850078" path="/var/lib/kubelet/pods/91301a32-1458-4112-b53e-5d5aa1850078/volumes"
Sep 15 21:14:00.526312 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.526264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:14:00.528922 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.528901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/806f2b57-c8d5-4011-85b8-3f61cee58e90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.528939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/806f2b57-c8d5-4011-85b8-3f61cee58e90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.528963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/806f2b57-c8d5-4011-85b8-3f61cee58e90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.528995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529163 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.529055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529163 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.529093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/806f2b57-c8d5-4011-85b8-3f61cee58e90-config-out\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529163 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.529120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529163 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.529147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-web-config\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529392 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.529183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-config-volume\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529392 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.529208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529392 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.529231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6kk\" (UniqueName: \"kubernetes.io/projected/806f2b57-c8d5-4011-85b8-3f61cee58e90-kube-api-access-zs6kk\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529392 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.529262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.529392 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.529282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806f2b57-c8d5-4011-85b8-3f61cee58e90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.629675 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/806f2b57-c8d5-4011-85b8-3f61cee58e90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.629772 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/806f2b57-c8d5-4011-85b8-3f61cee58e90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.629772 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/806f2b57-c8d5-4011-85b8-3f61cee58e90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.629772 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.629772 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.629772 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/806f2b57-c8d5-4011-85b8-3f61cee58e90-config-out\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.630006 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.630006 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-web-config\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.630006 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-config-volume\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.630006 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.630006 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6kk\" (UniqueName: \"kubernetes.io/projected/806f2b57-c8d5-4011-85b8-3f61cee58e90-kube-api-access-zs6kk\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.630006 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.630006 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.629965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806f2b57-c8d5-4011-85b8-3f61cee58e90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.630318 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.630083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/806f2b57-c8d5-4011-85b8-3f61cee58e90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.630659 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.630628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806f2b57-c8d5-4011-85b8-3f61cee58e90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.630741 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.630678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/806f2b57-c8d5-4011-85b8-3f61cee58e90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.632608 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.632565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/806f2b57-c8d5-4011-85b8-3f61cee58e90-config-out\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.632807 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.632766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-web-config\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.633089 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.633056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-config-volume\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.633292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.633273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.633383 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.633297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.633448 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.633429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/806f2b57-c8d5-4011-85b8-3f61cee58e90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.633545 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.633529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.633700 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.633675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.634234 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.634219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/806f2b57-c8d5-4011-85b8-3f61cee58e90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.639430 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.639401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6kk\" (UniqueName: \"kubernetes.io/projected/806f2b57-c8d5-4011-85b8-3f61cee58e90-kube-api-access-zs6kk\") pod \"alertmanager-main-0\" (UID: \"806f2b57-c8d5-4011-85b8-3f61cee58e90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:00.816476 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:00.816430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:14:01.090664 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:01.090643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:14:01.092559 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:14:01.092531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806f2b57_c8d5_4011_85b8_3f61cee58e90.slice/crio-07e85d91fce3d0e7499ab4b0d041999ea856bfd516e8dece657c36a3f1b28ef4 WatchSource:0}: Error finding container 07e85d91fce3d0e7499ab4b0d041999ea856bfd516e8dece657c36a3f1b28ef4: Status 404 returned error can't find the container with id 07e85d91fce3d0e7499ab4b0d041999ea856bfd516e8dece657c36a3f1b28ef4
Sep 15 21:14:01.450524 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:01.450495    2563 generic.go:356] "Generic (PLEG): container finished" podID="806f2b57-c8d5-4011-85b8-3f61cee58e90" containerID="3bc64f4034b3ab67e67151e65f6a00541e04a6580950a72ed0913dfb3981cd6d" exitCode=0
Sep 15 21:14:01.450654 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:01.450575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"806f2b57-c8d5-4011-85b8-3f61cee58e90","Type":"ContainerDied","Data":"3bc64f4034b3ab67e67151e65f6a00541e04a6580950a72ed0913dfb3981cd6d"}
Sep 15 21:14:01.450654 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:01.450632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"806f2b57-c8d5-4011-85b8-3f61cee58e90","Type":"ContainerStarted","Data":"07e85d91fce3d0e7499ab4b0d041999ea856bfd516e8dece657c36a3f1b28ef4"}
Sep 15 21:14:02.460146 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:02.460110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"806f2b57-c8d5-4011-85b8-3f61cee58e90","Type":"ContainerStarted","Data":"682e594cc8cc486e0e5d0efe37785a357ffe0884b7b74e5cf958217890dce131"}
Sep 15 21:14:02.460585 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:02.460199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"806f2b57-c8d5-4011-85b8-3f61cee58e90","Type":"ContainerStarted","Data":"8442e72d630b89408d5a4927a2b4003fffc6b97adc3f25ebec3e703752e152fe"}
Sep 15 21:14:02.460585 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:02.460228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"806f2b57-c8d5-4011-85b8-3f61cee58e90","Type":"ContainerStarted","Data":"c2d9e9d1450a963ecffef69a36b4b2785ba59571ca4201a50dfbf67e9ae1e17c"}
Sep 15 21:14:02.460585 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:02.460251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"806f2b57-c8d5-4011-85b8-3f61cee58e90","Type":"ContainerStarted","Data":"ee138b7ebf19fc2a9d9dd20e3175083618bf9b4b1d64c174bfd1bc4703a6c315"}
Sep 15 21:14:02.460585 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:02.460276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"806f2b57-c8d5-4011-85b8-3f61cee58e90","Type":"ContainerStarted","Data":"0cf788310146513b24e14c1ba0fa1fafcf438825186645411c521aabb71c5403"}
Sep 15 21:14:02.460585 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:02.460298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"806f2b57-c8d5-4011-85b8-3f61cee58e90","Type":"ContainerStarted","Data":"bd2c49638e228289cd62e21c307ec06de42317bf08ad95356f020904781d024e"}
Sep 15 21:14:02.487744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:02.487701    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.472666375 podStartE2EDuration="2.487685333s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.018958ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:02.485392991 +0000 UTC m=+232.448450370" watchObservedRunningTime="2026-09-15 21:14:02.487685333 +0000 UTC m=+232.450742715"
Sep 15 21:14:40.966689 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:40.966654    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64bcffdbf-zmg9f"]
Sep 15 21:14:40.969104 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:40.969087    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:40.979279 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:40.979254    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bcffdbf-zmg9f"]
Sep 15 21:14:41.039612 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.039583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-oauth-serving-cert\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.039739 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.039657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-service-ca\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.039739 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.039692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-oauth-config\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.039739 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.039710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-trusted-ca-bundle\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.039739 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.039737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvbg\" (UniqueName: \"kubernetes.io/projected/4a92efa3-20e1-458b-841f-2dff8843b882-kube-api-access-qnvbg\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.039914 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.039797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-console-config\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.039914 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.039844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-serving-cert\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.140655 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.140621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-serving-cert\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.140824 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.140663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-oauth-serving-cert\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.140824 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.140716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-service-ca\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.140824 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.140752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-oauth-config\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.140824 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.140776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-trusted-ca-bundle\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.141042 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.140858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvbg\" (UniqueName: \"kubernetes.io/projected/4a92efa3-20e1-458b-841f-2dff8843b882-kube-api-access-qnvbg\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.141042 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.140908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-console-config\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.141554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.141524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-oauth-serving-cert\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.141728 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.141698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-service-ca\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.141835 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.141733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-trusted-ca-bundle\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.141835 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.141733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-console-config\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.143144 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.143125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-serving-cert\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.143230 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.143145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-oauth-config\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.149531 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.149509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvbg\" (UniqueName: \"kubernetes.io/projected/4a92efa3-20e1-458b-841f-2dff8843b882-kube-api-access-qnvbg\") pod \"console-64bcffdbf-zmg9f\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") " pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.279066 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.279000    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:41.546798 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.546722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bcffdbf-zmg9f"]
Sep 15 21:14:41.549515 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:14:41.549489    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a92efa3_20e1_458b_841f_2dff8843b882.slice/crio-38d1f6b91036e5bf43ebe63d226022de0cc4930701265e6533568c2bd656902a WatchSource:0}: Error finding container 38d1f6b91036e5bf43ebe63d226022de0cc4930701265e6533568c2bd656902a: Status 404 returned error can't find the container with id 38d1f6b91036e5bf43ebe63d226022de0cc4930701265e6533568c2bd656902a
Sep 15 21:14:41.612626 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.612604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bcffdbf-zmg9f" event={"ID":"4a92efa3-20e1-458b-841f-2dff8843b882","Type":"ContainerStarted","Data":"40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a"}
Sep 15 21:14:41.612726 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.612643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bcffdbf-zmg9f" event={"ID":"4a92efa3-20e1-458b-841f-2dff8843b882","Type":"ContainerStarted","Data":"38d1f6b91036e5bf43ebe63d226022de0cc4930701265e6533568c2bd656902a"}
Sep 15 21:14:41.630803 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:41.630759    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64bcffdbf-zmg9f" podStartSLOduration=1.630742309 podStartE2EDuration="1.630742309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:41.629487177 +0000 UTC m=+271.592544558" watchObservedRunningTime="2026-09-15 21:14:41.630742309 +0000 UTC m=+271.593799689"
Sep 15 21:14:51.279849 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:51.279816    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:51.280303 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:51.280135    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:51.284546 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:51.284523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:51.652836 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:51.652815    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:14:51.699529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:14:51.699503    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c59bb7d79-t6skv"]
Sep 15 21:15:10.450639 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:10.450609    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 21:15:16.717778 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.717721    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c59bb7d79-t6skv" podUID="dc65c4e6-ee38-438c-a23b-5150746d54ef" containerName="console" containerID="cri-o://ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542" gracePeriod=15
Sep 15 21:15:16.952058 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.952034    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:15:16.998284 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998215    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-serving-cert\") pod \"dc65c4e6-ee38-438c-a23b-5150746d54ef\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") "
Sep 15 21:15:16.998432 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998296    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-config\") pod \"dc65c4e6-ee38-438c-a23b-5150746d54ef\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") "
Sep 15 21:15:16.998432 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-service-ca\") pod \"dc65c4e6-ee38-438c-a23b-5150746d54ef\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") "
Sep 15 21:15:16.998432 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998363    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-trusted-ca-bundle\") pod \"dc65c4e6-ee38-438c-a23b-5150746d54ef\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") "
Sep 15 21:15:16.998432 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998397    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj7gr\" (UniqueName: \"kubernetes.io/projected/dc65c4e6-ee38-438c-a23b-5150746d54ef-kube-api-access-bj7gr\") pod \"dc65c4e6-ee38-438c-a23b-5150746d54ef\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") "
Sep 15 21:15:16.998432 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998419    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-oauth-config\") pod \"dc65c4e6-ee38-438c-a23b-5150746d54ef\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") "
Sep 15 21:15:16.998678 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998453    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-oauth-serving-cert\") pod \"dc65c4e6-ee38-438c-a23b-5150746d54ef\" (UID: \"dc65c4e6-ee38-438c-a23b-5150746d54ef\") "
Sep 15 21:15:16.998736 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998690    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-config" (OuterVolumeSpecName: "console-config") pod "dc65c4e6-ee38-438c-a23b-5150746d54ef" (UID: "dc65c4e6-ee38-438c-a23b-5150746d54ef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:15:16.998736 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998719    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-service-ca" (OuterVolumeSpecName: "service-ca") pod "dc65c4e6-ee38-438c-a23b-5150746d54ef" (UID: "dc65c4e6-ee38-438c-a23b-5150746d54ef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:15:16.998869 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998832    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dc65c4e6-ee38-438c-a23b-5150746d54ef" (UID: "dc65c4e6-ee38-438c-a23b-5150746d54ef"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:15:16.999000 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:16.998976    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dc65c4e6-ee38-438c-a23b-5150746d54ef" (UID: "dc65c4e6-ee38-438c-a23b-5150746d54ef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:15:17.000256 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.000234    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dc65c4e6-ee38-438c-a23b-5150746d54ef" (UID: "dc65c4e6-ee38-438c-a23b-5150746d54ef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:15:17.000543 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.000520    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dc65c4e6-ee38-438c-a23b-5150746d54ef" (UID: "dc65c4e6-ee38-438c-a23b-5150746d54ef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:15:17.000634 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.000551    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc65c4e6-ee38-438c-a23b-5150746d54ef-kube-api-access-bj7gr" (OuterVolumeSpecName: "kube-api-access-bj7gr") pod "dc65c4e6-ee38-438c-a23b-5150746d54ef" (UID: "dc65c4e6-ee38-438c-a23b-5150746d54ef"). InnerVolumeSpecName "kube-api-access-bj7gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:15:17.098994 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.098973    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-serving-cert\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:17.098994 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.098993    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-config\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:17.099112 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.099002    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-service-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:17.099112 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.099011    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-trusted-ca-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:17.099112 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.099019    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7gr\" (UniqueName: \"kubernetes.io/projected/dc65c4e6-ee38-438c-a23b-5150746d54ef-kube-api-access-bj7gr\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:17.099112 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.099029    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc65c4e6-ee38-438c-a23b-5150746d54ef-console-oauth-config\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:17.099112 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.099037    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc65c4e6-ee38-438c-a23b-5150746d54ef-oauth-serving-cert\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:17.750125 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.750097    2563 generic.go:356] "Generic (PLEG): container finished" podID="dc65c4e6-ee38-438c-a23b-5150746d54ef" containerID="ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542" exitCode=0
Sep 15 21:15:17.750527 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.750132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c59bb7d79-t6skv" event={"ID":"dc65c4e6-ee38-438c-a23b-5150746d54ef","Type":"ContainerDied","Data":"ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542"}
Sep 15 21:15:17.750527 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.750135    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c59bb7d79-t6skv"
Sep 15 21:15:17.750527 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.750178    2563 scope.go:117] "RemoveContainer" containerID="ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542"
Sep 15 21:15:17.750527 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.750169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c59bb7d79-t6skv" event={"ID":"dc65c4e6-ee38-438c-a23b-5150746d54ef","Type":"ContainerDied","Data":"79464c3904bab81b835e62fde01afa7bd40d95c07136264cc82fb1f264b33983"}
Sep 15 21:15:17.758039 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.758017    2563 scope.go:117] "RemoveContainer" containerID="ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542"
Sep 15 21:15:17.758306 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:15:17.758285    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542\": container with ID starting with ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542 not found: ID does not exist" containerID="ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542"
Sep 15 21:15:17.758413 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.758313    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542"} err="failed to get container status \"ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542\": rpc error: code = NotFound desc = could not find container \"ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542\": container with ID starting with ea8e3422b2b77dbc5c9f35206e382b49082808ee19c33595829523c408fe9542 not found: ID does not exist"
Sep 15 21:15:17.771102 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.771084    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c59bb7d79-t6skv"]
Sep 15 21:15:17.774577 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.774556    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c59bb7d79-t6skv"]
Sep 15 21:15:17.864636 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.864613    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"]
Sep 15 21:15:17.864912 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.864901    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc65c4e6-ee38-438c-a23b-5150746d54ef" containerName="console"
Sep 15 21:15:17.864957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.864914    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc65c4e6-ee38-438c-a23b-5150746d54ef" containerName="console"
Sep 15 21:15:17.864993 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.864961    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc65c4e6-ee38-438c-a23b-5150746d54ef" containerName="console"
Sep 15 21:15:17.867506 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.867492    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:17.869754 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.869737    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kjpxg\""
Sep 15 21:15:17.869754 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.869745    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 21:15:17.869867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.869840    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 21:15:17.877132 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.877115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"]
Sep 15 21:15:17.905730 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.905706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:17.905854 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.905822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcp4v\" (UniqueName: \"kubernetes.io/projected/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-kube-api-access-hcp4v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:17.905958 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:17.905871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:18.006423 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.006372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcp4v\" (UniqueName: \"kubernetes.io/projected/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-kube-api-access-hcp4v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:18.006423 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.006407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:18.006577 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.006437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:18.006725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.006711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:18.006779 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.006743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:18.015206 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.015180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcp4v\" (UniqueName: \"kubernetes.io/projected/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-kube-api-access-hcp4v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:18.176408 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.176388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:18.448654 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.448630    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"]
Sep 15 21:15:18.452277 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:15:18.452252    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c45e112_528f_4e18_a83c_eb0b5ef8f2bf.slice/crio-798a4d3d8a80a56368eaee31ed70df392285b6dde835f94d0736429fff9b8e58 WatchSource:0}: Error finding container 798a4d3d8a80a56368eaee31ed70df392285b6dde835f94d0736429fff9b8e58: Status 404 returned error can't find the container with id 798a4d3d8a80a56368eaee31ed70df392285b6dde835f94d0736429fff9b8e58
Sep 15 21:15:18.454555 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.454540    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:15:18.524401 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.524379    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc65c4e6-ee38-438c-a23b-5150746d54ef" path="/var/lib/kubelet/pods/dc65c4e6-ee38-438c-a23b-5150746d54ef/volumes"
Sep 15 21:15:18.754766 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:18.754717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk" event={"ID":"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf","Type":"ContainerStarted","Data":"798a4d3d8a80a56368eaee31ed70df392285b6dde835f94d0736429fff9b8e58"}
Sep 15 21:15:23.782819 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:23.782783    2563 generic.go:356] "Generic (PLEG): container finished" podID="1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" containerID="f53f03ffb91c8bd34d58fa0e2d029fa42f7a17124fc3567bcbea5e1153f24d98" exitCode=0
Sep 15 21:15:23.783149 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:23.782854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk" event={"ID":"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf","Type":"ContainerDied","Data":"f53f03ffb91c8bd34d58fa0e2d029fa42f7a17124fc3567bcbea5e1153f24d98"}
Sep 15 21:15:26.796898 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:26.796801    2563 generic.go:356] "Generic (PLEG): container finished" podID="1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" containerID="8936875c74d0cae40c3e98d91069dcdaec9c3e35e36c5aacbaa45d625eb92a62" exitCode=0
Sep 15 21:15:26.797243 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:26.796893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk" event={"ID":"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf","Type":"ContainerDied","Data":"8936875c74d0cae40c3e98d91069dcdaec9c3e35e36c5aacbaa45d625eb92a62"}
Sep 15 21:15:33.836279 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:33.836248    2563 generic.go:356] "Generic (PLEG): container finished" podID="1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" containerID="c19dfdabddb9831dbdba04627c4e5c995211fe49d46bb9233cde36f8b24068c4" exitCode=0
Sep 15 21:15:33.836605 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:33.836336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk" event={"ID":"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf","Type":"ContainerDied","Data":"c19dfdabddb9831dbdba04627c4e5c995211fe49d46bb9233cde36f8b24068c4"}
Sep 15 21:15:34.963754 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:34.963728    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:35.036545 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.036517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcp4v\" (UniqueName: \"kubernetes.io/projected/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-kube-api-access-hcp4v\") pod \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") "
Sep 15 21:15:35.036665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.036599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-bundle\") pod \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") "
Sep 15 21:15:35.036665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.036645    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-util\") pod \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\" (UID: \"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf\") "
Sep 15 21:15:35.037363 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.037320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-bundle" (OuterVolumeSpecName: "bundle") pod "1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" (UID: "1c45e112-528f-4e18-a83c-eb0b5ef8f2bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:35.038562 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.038537    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-kube-api-access-hcp4v" (OuterVolumeSpecName: "kube-api-access-hcp4v") pod "1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" (UID: "1c45e112-528f-4e18-a83c-eb0b5ef8f2bf"). InnerVolumeSpecName "kube-api-access-hcp4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:15:35.040791 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.040771    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-util" (OuterVolumeSpecName: "util") pod "1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" (UID: "1c45e112-528f-4e18-a83c-eb0b5ef8f2bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:35.137867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.137847    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.137867 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.137865    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-util\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.137979 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.137874    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcp4v\" (UniqueName: \"kubernetes.io/projected/1c45e112-528f-4e18-a83c-eb0b5ef8f2bf-kube-api-access-hcp4v\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:35.845310 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.845282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk" event={"ID":"1c45e112-528f-4e18-a83c-eb0b5ef8f2bf","Type":"ContainerDied","Data":"798a4d3d8a80a56368eaee31ed70df392285b6dde835f94d0736429fff9b8e58"}
Sep 15 21:15:35.845425 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.845320    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drfzk"
Sep 15 21:15:35.845425 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:35.845325    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="798a4d3d8a80a56368eaee31ed70df392285b6dde835f94d0736429fff9b8e58"
Sep 15 21:15:41.052104 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.052072    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"]
Sep 15 21:15:41.052482 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.052414    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" containerName="pull"
Sep 15 21:15:41.052482 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.052425    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" containerName="pull"
Sep 15 21:15:41.052482 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.052432    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" containerName="extract"
Sep 15 21:15:41.052482 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.052437    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" containerName="extract"
Sep 15 21:15:41.052482 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.052444    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" containerName="util"
Sep 15 21:15:41.052482 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.052449    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" containerName="util"
Sep 15 21:15:41.052698 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.052496    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c45e112-528f-4e18-a83c-eb0b5ef8f2bf" containerName="extract"
Sep 15 21:15:41.055816 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.055798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"
Sep 15 21:15:41.058070 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.058042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qt5pp\""
Sep 15 21:15:41.058070 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.058051    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:15:41.058214 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.058053    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 15 21:15:41.066781 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.066763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"]
Sep 15 21:15:41.183407 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.183380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhbhw\" (UniqueName: \"kubernetes.io/projected/bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d-kube-api-access-zhbhw\") pod \"cert-manager-operator-controller-manager-55c444bf86-7dss5\" (UID: \"bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"
Sep 15 21:15:41.183517 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.183440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7dss5\" (UID: \"bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"
Sep 15 21:15:41.284133 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.284109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7dss5\" (UID: \"bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"
Sep 15 21:15:41.284223 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.284149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhbhw\" (UniqueName: \"kubernetes.io/projected/bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d-kube-api-access-zhbhw\") pod \"cert-manager-operator-controller-manager-55c444bf86-7dss5\" (UID: \"bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"
Sep 15 21:15:41.284498 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.284479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7dss5\" (UID: \"bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"
Sep 15 21:15:41.292401 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.292384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhbhw\" (UniqueName: \"kubernetes.io/projected/bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d-kube-api-access-zhbhw\") pod \"cert-manager-operator-controller-manager-55c444bf86-7dss5\" (UID: \"bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"
Sep 15 21:15:41.364120 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.364077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"
Sep 15 21:15:41.637556 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.637532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5"]
Sep 15 21:15:41.640448 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:15:41.640425    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf327b8e_8cd5_4a0b_a5b4_e3e49b43944d.slice/crio-8a148dc787a0fc086e6bbb39e6d190d17a3041309ecfb6ccd5ad0c61996d2b83 WatchSource:0}: Error finding container 8a148dc787a0fc086e6bbb39e6d190d17a3041309ecfb6ccd5ad0c61996d2b83: Status 404 returned error can't find the container with id 8a148dc787a0fc086e6bbb39e6d190d17a3041309ecfb6ccd5ad0c61996d2b83
Sep 15 21:15:41.871874 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:41.871844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5" event={"ID":"bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d","Type":"ContainerStarted","Data":"8a148dc787a0fc086e6bbb39e6d190d17a3041309ecfb6ccd5ad0c61996d2b83"}
Sep 15 21:15:43.883480 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:43.883448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5" event={"ID":"bf327b8e-8cd5-4a0b-a5b4-e3e49b43944d","Type":"ContainerStarted","Data":"420c5b8d1bb27942966a07f4fb3199eb768ddc56618c305d1054d406fe124d7f"}
Sep 15 21:15:43.904789 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:43.904743    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7dss5" podStartSLOduration=1.4841166000000001 podStartE2EDuration="2.904728552s" totalImagesPullingTime="1.420611952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:43.903995436 +0000 UTC m=+333.867052829" watchObservedRunningTime="2026-09-15 21:15:43.904728552 +0000 UTC m=+333.867785932"
Sep 15 21:15:45.110398 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.110366    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hk4mt"]
Sep 15 21:15:45.113656 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.113640    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt"
Sep 15 21:15:45.117257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.117236    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 21:15:45.117403 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.117385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tfcs6\""
Sep 15 21:15:45.118143 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.118129    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 21:15:45.129989 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.129967    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hk4mt"]
Sep 15 21:15:45.215056 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.215032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln5ch\" (UniqueName: \"kubernetes.io/projected/876a6e19-bdb5-43a8-b102-ce2ae3eed266-kube-api-access-ln5ch\") pod \"cert-manager-webhook-6486ffd997-hk4mt\" (UID: \"876a6e19-bdb5-43a8-b102-ce2ae3eed266\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt"
Sep 15 21:15:45.215177 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.215075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/876a6e19-bdb5-43a8-b102-ce2ae3eed266-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hk4mt\" (UID: \"876a6e19-bdb5-43a8-b102-ce2ae3eed266\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt"
Sep 15 21:15:45.247358 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.247329    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"]
Sep 15 21:15:45.250748 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.250733    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.254308 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.254292    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 21:15:45.254581 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.254560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kjpxg\""
Sep 15 21:15:45.255071 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.255053    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 21:15:45.312780 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.312754    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"]
Sep 15 21:15:45.315575 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.315548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdptl\" (UniqueName: \"kubernetes.io/projected/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-kube-api-access-tdptl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.315685 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.315590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln5ch\" (UniqueName: \"kubernetes.io/projected/876a6e19-bdb5-43a8-b102-ce2ae3eed266-kube-api-access-ln5ch\") pod \"cert-manager-webhook-6486ffd997-hk4mt\" (UID: \"876a6e19-bdb5-43a8-b102-ce2ae3eed266\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt"
Sep 15 21:15:45.315685 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.315615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.315787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.315708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/876a6e19-bdb5-43a8-b102-ce2ae3eed266-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hk4mt\" (UID: \"876a6e19-bdb5-43a8-b102-ce2ae3eed266\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt"
Sep 15 21:15:45.315842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.315787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.324153 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.324134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln5ch\" (UniqueName: \"kubernetes.io/projected/876a6e19-bdb5-43a8-b102-ce2ae3eed266-kube-api-access-ln5ch\") pod \"cert-manager-webhook-6486ffd997-hk4mt\" (UID: \"876a6e19-bdb5-43a8-b102-ce2ae3eed266\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt"
Sep 15 21:15:45.324241 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.324171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/876a6e19-bdb5-43a8-b102-ce2ae3eed266-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hk4mt\" (UID: \"876a6e19-bdb5-43a8-b102-ce2ae3eed266\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt"
Sep 15 21:15:45.416620 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.416600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.416708 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.416635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdptl\" (UniqueName: \"kubernetes.io/projected/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-kube-api-access-tdptl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.416708 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.416660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.416913 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.416896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.416974 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.416959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.427777 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.427758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdptl\" (UniqueName: \"kubernetes.io/projected/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-kube-api-access-tdptl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.439588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.439569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt"
Sep 15 21:15:45.559895 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.559874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:45.716185 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.716164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hk4mt"]
Sep 15 21:15:45.718103 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:15:45.718073    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod876a6e19_bdb5_43a8_b102_ce2ae3eed266.slice/crio-62493fdb9a68c3ff53e38dcab5eedb115d2613b7f12872d8f7b5af0a1feebeab WatchSource:0}: Error finding container 62493fdb9a68c3ff53e38dcab5eedb115d2613b7f12872d8f7b5af0a1feebeab: Status 404 returned error can't find the container with id 62493fdb9a68c3ff53e38dcab5eedb115d2613b7f12872d8f7b5af0a1feebeab
Sep 15 21:15:45.827699 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.827676    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"]
Sep 15 21:15:45.829471 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:15:45.829442    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0d1d73_fac0_4469_ae6b_f8e8f0ce8a84.slice/crio-f0de961ddf229ee7a8ac594a66378194ca1958f04ebed1b26abe5d98e4f24de1 WatchSource:0}: Error finding container f0de961ddf229ee7a8ac594a66378194ca1958f04ebed1b26abe5d98e4f24de1: Status 404 returned error can't find the container with id f0de961ddf229ee7a8ac594a66378194ca1958f04ebed1b26abe5d98e4f24de1
Sep 15 21:15:45.899669 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.899642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx" event={"ID":"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84","Type":"ContainerStarted","Data":"f0de961ddf229ee7a8ac594a66378194ca1958f04ebed1b26abe5d98e4f24de1"}
Sep 15 21:15:45.900868 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:45.900849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt" event={"ID":"876a6e19-bdb5-43a8-b102-ce2ae3eed266","Type":"ContainerStarted","Data":"62493fdb9a68c3ff53e38dcab5eedb115d2613b7f12872d8f7b5af0a1feebeab"}
Sep 15 21:15:46.907912 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:46.907856    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" containerID="f89028a489f847e592d21c9cec92d96107a71ac1cc06747d8be205eb6df0d5a5" exitCode=0
Sep 15 21:15:46.908379 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:46.907946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx" event={"ID":"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84","Type":"ContainerDied","Data":"f89028a489f847e592d21c9cec92d96107a71ac1cc06747d8be205eb6df0d5a5"}
Sep 15 21:15:47.115573 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.115539    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"]
Sep 15 21:15:47.119138 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.119118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"
Sep 15 21:15:47.121406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.121383    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-wcwqf\""
Sep 15 21:15:47.129082 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.129009    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"]
Sep 15 21:15:47.230787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.230755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9pn\" (UniqueName: \"kubernetes.io/projected/37bd44cd-7d15-4625-bc27-64679ad92635-kube-api-access-sz9pn\") pod \"cert-manager-cainjector-559cdd8c87-4k56d\" (UID: \"37bd44cd-7d15-4625-bc27-64679ad92635\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"
Sep 15 21:15:47.230920 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.230799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37bd44cd-7d15-4625-bc27-64679ad92635-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4k56d\" (UID: \"37bd44cd-7d15-4625-bc27-64679ad92635\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"
Sep 15 21:15:47.331592 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.331554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9pn\" (UniqueName: \"kubernetes.io/projected/37bd44cd-7d15-4625-bc27-64679ad92635-kube-api-access-sz9pn\") pod \"cert-manager-cainjector-559cdd8c87-4k56d\" (UID: \"37bd44cd-7d15-4625-bc27-64679ad92635\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"
Sep 15 21:15:47.331765 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.331610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37bd44cd-7d15-4625-bc27-64679ad92635-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4k56d\" (UID: \"37bd44cd-7d15-4625-bc27-64679ad92635\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"
Sep 15 21:15:47.341415 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.341376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37bd44cd-7d15-4625-bc27-64679ad92635-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4k56d\" (UID: \"37bd44cd-7d15-4625-bc27-64679ad92635\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"
Sep 15 21:15:47.341615 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.341592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9pn\" (UniqueName: \"kubernetes.io/projected/37bd44cd-7d15-4625-bc27-64679ad92635-kube-api-access-sz9pn\") pod \"cert-manager-cainjector-559cdd8c87-4k56d\" (UID: \"37bd44cd-7d15-4625-bc27-64679ad92635\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"
Sep 15 21:15:47.430957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.430927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"
Sep 15 21:15:47.710322 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.710297    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4k56d"]
Sep 15 21:15:47.712307 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:15:47.712278    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37bd44cd_7d15_4625_bc27_64679ad92635.slice/crio-d7a4cc5dd5a938ff1e4d9c8f32122221b538fd44f09550ae7749a481854a3458 WatchSource:0}: Error finding container d7a4cc5dd5a938ff1e4d9c8f32122221b538fd44f09550ae7749a481854a3458: Status 404 returned error can't find the container with id d7a4cc5dd5a938ff1e4d9c8f32122221b538fd44f09550ae7749a481854a3458
Sep 15 21:15:47.914189 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:47.914153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d" event={"ID":"37bd44cd-7d15-4625-bc27-64679ad92635","Type":"ContainerStarted","Data":"d7a4cc5dd5a938ff1e4d9c8f32122221b538fd44f09550ae7749a481854a3458"}
Sep 15 21:15:48.922375 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:48.922261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt" event={"ID":"876a6e19-bdb5-43a8-b102-ce2ae3eed266","Type":"ContainerStarted","Data":"119e0db0b9fb493ac9dd12cf54ad1f50abb268b0e3e675fe0d6939e4c98318ba"}
Sep 15 21:15:48.922812 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:48.922499    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt"
Sep 15 21:15:48.924463 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:48.924423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d" event={"ID":"37bd44cd-7d15-4625-bc27-64679ad92635","Type":"ContainerStarted","Data":"b92d828b9dd728f2e762e1b4ba4834bee4dfd6c6544b129f448681ea90bd4db7"}
Sep 15 21:15:48.940566 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:48.940510    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt" podStartSLOduration=0.909019758 podStartE2EDuration="3.940491128s" totalImagesPullingTime="3.03147137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:48.9395398 +0000 UTC m=+338.902597184" watchObservedRunningTime="2026-09-15 21:15:48.940491128 +0000 UTC m=+338.903548509"
Sep 15 21:15:48.958595 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:48.958553    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4k56d" podStartSLOduration=0.920830777 podStartE2EDuration="1.958540068s" totalImagesPullingTime="1.037709291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:48.957056123 +0000 UTC m=+338.920113504" watchObservedRunningTime="2026-09-15 21:15:48.958540068 +0000 UTC m=+338.921597448"
Sep 15 21:15:50.935472 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:50.935438    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" containerID="c01683a253ba36ac0f55108cec50517a58fe3d852384a9c27323015fc168941d" exitCode=0
Sep 15 21:15:50.935888 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:50.935517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx" event={"ID":"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84","Type":"ContainerDied","Data":"c01683a253ba36ac0f55108cec50517a58fe3d852384a9c27323015fc168941d"}
Sep 15 21:15:51.942024 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:51.941993    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" containerID="f5188f0d5156b7fe9ce312745a376b52d5583b9bd6c53ba34a3a1719ba4a2f3a" exitCode=0
Sep 15 21:15:51.942459 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:51.942075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx" event={"ID":"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84","Type":"ContainerDied","Data":"f5188f0d5156b7fe9ce312745a376b52d5583b9bd6c53ba34a3a1719ba4a2f3a"}
Sep 15 21:15:53.064206 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.064183    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:53.179942 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.179910    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdptl\" (UniqueName: \"kubernetes.io/projected/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-kube-api-access-tdptl\") pod \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") "
Sep 15 21:15:53.180090 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.179983    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-util\") pod \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") "
Sep 15 21:15:53.180090 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.180007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-bundle\") pod \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\" (UID: \"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84\") "
Sep 15 21:15:53.180436 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.180409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-bundle" (OuterVolumeSpecName: "bundle") pod "cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" (UID: "cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:53.181994 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.181966    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-kube-api-access-tdptl" (OuterVolumeSpecName: "kube-api-access-tdptl") pod "cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" (UID: "cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84"). InnerVolumeSpecName "kube-api-access-tdptl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:15:53.183899 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.183879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-util" (OuterVolumeSpecName: "util") pod "cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" (UID: "cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:53.280957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.280907    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdptl\" (UniqueName: \"kubernetes.io/projected/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-kube-api-access-tdptl\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:53.280957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.280927    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-util\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:53.280957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.280936    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:15:53.952451 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.952421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx" event={"ID":"cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84","Type":"ContainerDied","Data":"f0de961ddf229ee7a8ac594a66378194ca1958f04ebed1b26abe5d98e4f24de1"}
Sep 15 21:15:53.952451 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.952443    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9prjx"
Sep 15 21:15:53.952588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:53.952470    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0de961ddf229ee7a8ac594a66378194ca1958f04ebed1b26abe5d98e4f24de1"
Sep 15 21:15:54.931223 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:15:54.931203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hk4mt"
Sep 15 21:16:03.346065 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.345989    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"]
Sep 15 21:16:03.346508 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.346491    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" containerName="extract"
Sep 15 21:16:03.346593 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.346511    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" containerName="extract"
Sep 15 21:16:03.346593 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.346530    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" containerName="pull"
Sep 15 21:16:03.346593 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.346538    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" containerName="pull"
Sep 15 21:16:03.346593 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.346574    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" containerName="util"
Sep 15 21:16:03.346593 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.346583    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" containerName="util"
Sep 15 21:16:03.346968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.346659    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf0d1d73-fac0-4469-ae6b-f8e8f0ce8a84" containerName="extract"
Sep 15 21:16:03.349508 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.349488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"
Sep 15 21:16:03.351967 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.351944    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-t6ns5\""
Sep 15 21:16:03.351967 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.351963    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 21:16:03.352132 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.352004    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:16:03.358851 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.358833    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"]
Sep 15 21:16:03.454236 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.454215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/589b7486-5e05-42d2-8e2b-98a33276d7f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7bjv8\" (UID: \"589b7486-5e05-42d2-8e2b-98a33276d7f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"
Sep 15 21:16:03.454321 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.454270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhgbf\" (UniqueName: \"kubernetes.io/projected/589b7486-5e05-42d2-8e2b-98a33276d7f0-kube-api-access-dhgbf\") pod \"openshift-lws-operator-fd8ccff4c-7bjv8\" (UID: \"589b7486-5e05-42d2-8e2b-98a33276d7f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"
Sep 15 21:16:03.554610 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.554586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/589b7486-5e05-42d2-8e2b-98a33276d7f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7bjv8\" (UID: \"589b7486-5e05-42d2-8e2b-98a33276d7f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"
Sep 15 21:16:03.554711 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.554640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhgbf\" (UniqueName: \"kubernetes.io/projected/589b7486-5e05-42d2-8e2b-98a33276d7f0-kube-api-access-dhgbf\") pod \"openshift-lws-operator-fd8ccff4c-7bjv8\" (UID: \"589b7486-5e05-42d2-8e2b-98a33276d7f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"
Sep 15 21:16:03.554942 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.554921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/589b7486-5e05-42d2-8e2b-98a33276d7f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7bjv8\" (UID: \"589b7486-5e05-42d2-8e2b-98a33276d7f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"
Sep 15 21:16:03.563134 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.563114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhgbf\" (UniqueName: \"kubernetes.io/projected/589b7486-5e05-42d2-8e2b-98a33276d7f0-kube-api-access-dhgbf\") pod \"openshift-lws-operator-fd8ccff4c-7bjv8\" (UID: \"589b7486-5e05-42d2-8e2b-98a33276d7f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"
Sep 15 21:16:03.658567 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.658548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"
Sep 15 21:16:03.942241 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.942219    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8"]
Sep 15 21:16:03.944177 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:16:03.944152    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod589b7486_5e05_42d2_8e2b_98a33276d7f0.slice/crio-c50785b5cd4023d90c626630abd507ac61e60cf55b13d24c7d4c7aa702ee7f00 WatchSource:0}: Error finding container c50785b5cd4023d90c626630abd507ac61e60cf55b13d24c7d4c7aa702ee7f00: Status 404 returned error can't find the container with id c50785b5cd4023d90c626630abd507ac61e60cf55b13d24c7d4c7aa702ee7f00
Sep 15 21:16:03.996038 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:03.996007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8" event={"ID":"589b7486-5e05-42d2-8e2b-98a33276d7f0","Type":"ContainerStarted","Data":"c50785b5cd4023d90c626630abd507ac61e60cf55b13d24c7d4c7aa702ee7f00"}
Sep 15 21:16:06.007929 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:06.007895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8" event={"ID":"589b7486-5e05-42d2-8e2b-98a33276d7f0","Type":"ContainerStarted","Data":"9adcacef0560d1d23201a5bcaddbeafe3ca6b4ce2129c1ce8457cdaed5aac8fb"}
Sep 15 21:16:06.024048 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:06.024006    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bjv8" podStartSLOduration=1.13851339 podStartE2EDuration="3.023991852s" totalImagesPullingTime="1.885478462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:06.022693323 +0000 UTC m=+355.985750703" watchObservedRunningTime="2026-09-15 21:16:06.023991852 +0000 UTC m=+355.987049233"
Sep 15 21:16:10.725649 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.725579    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"]
Sep 15 21:16:10.728714 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.728694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:10.730910 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.730890    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 21:16:10.731017 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.730935    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kjpxg\""
Sep 15 21:16:10.731779 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.731766    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 21:16:10.737289 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.737266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"]
Sep 15 21:16:10.815139 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.815112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:10.815253 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.815143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrcl\" (UniqueName: \"kubernetes.io/projected/604b4550-7ea6-4b6d-aa30-767cf0b89a27-kube-api-access-svrcl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:10.815253 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.815202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:10.915654 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.915627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:10.915771 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.915688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:10.915771 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.915717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrcl\" (UniqueName: \"kubernetes.io/projected/604b4550-7ea6-4b6d-aa30-767cf0b89a27-kube-api-access-svrcl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:10.916033 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.916016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:10.916093 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.916044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:10.923874 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:10.923851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrcl\" (UniqueName: \"kubernetes.io/projected/604b4550-7ea6-4b6d-aa30-767cf0b89a27-kube-api-access-svrcl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:11.037965 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:11.037892    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:11.304967 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:11.304946    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"]
Sep 15 21:16:11.306486 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:16:11.306466    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604b4550_7ea6_4b6d_aa30_767cf0b89a27.slice/crio-aceb176e65ef6d6185d96c59515f3ee947f1d0fc1697abf7af7b2ccbcae60cbb WatchSource:0}: Error finding container aceb176e65ef6d6185d96c59515f3ee947f1d0fc1697abf7af7b2ccbcae60cbb: Status 404 returned error can't find the container with id aceb176e65ef6d6185d96c59515f3ee947f1d0fc1697abf7af7b2ccbcae60cbb
Sep 15 21:16:12.036446 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:12.036414    2563 generic.go:356] "Generic (PLEG): container finished" podID="604b4550-7ea6-4b6d-aa30-767cf0b89a27" containerID="6f605fecafb15fad289fa95bb1cbd26329196ed10ded4dcf62244e0cfdbedc17" exitCode=0
Sep 15 21:16:12.036764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:12.036500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x" event={"ID":"604b4550-7ea6-4b6d-aa30-767cf0b89a27","Type":"ContainerDied","Data":"6f605fecafb15fad289fa95bb1cbd26329196ed10ded4dcf62244e0cfdbedc17"}
Sep 15 21:16:12.036764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:12.036545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x" event={"ID":"604b4550-7ea6-4b6d-aa30-767cf0b89a27","Type":"ContainerStarted","Data":"aceb176e65ef6d6185d96c59515f3ee947f1d0fc1697abf7af7b2ccbcae60cbb"}
Sep 15 21:16:13.045810 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:13.045781    2563 generic.go:356] "Generic (PLEG): container finished" podID="604b4550-7ea6-4b6d-aa30-767cf0b89a27" containerID="6c6cae779942aab82828b06477b5d4a2e0ced466e0692eda853306372151f061" exitCode=0
Sep 15 21:16:13.046187 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:13.045864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x" event={"ID":"604b4550-7ea6-4b6d-aa30-767cf0b89a27","Type":"ContainerDied","Data":"6c6cae779942aab82828b06477b5d4a2e0ced466e0692eda853306372151f061"}
Sep 15 21:16:14.052999 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:14.052970    2563 generic.go:356] "Generic (PLEG): container finished" podID="604b4550-7ea6-4b6d-aa30-767cf0b89a27" containerID="318f8bbe3f0cc54e6e9144b99c62f1d8d09e4192e8945fea7a9b9e3ff3818ed0" exitCode=0
Sep 15 21:16:14.053314 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:14.053054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x" event={"ID":"604b4550-7ea6-4b6d-aa30-767cf0b89a27","Type":"ContainerDied","Data":"318f8bbe3f0cc54e6e9144b99c62f1d8d09e4192e8945fea7a9b9e3ff3818ed0"}
Sep 15 21:16:15.176730 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:15.176700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:15.353936 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:15.353427    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-bundle\") pod \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") "
Sep 15 21:16:15.353936 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:15.353473    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svrcl\" (UniqueName: \"kubernetes.io/projected/604b4550-7ea6-4b6d-aa30-767cf0b89a27-kube-api-access-svrcl\") pod \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") "
Sep 15 21:16:15.353936 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:15.353503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-util\") pod \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\" (UID: \"604b4550-7ea6-4b6d-aa30-767cf0b89a27\") "
Sep 15 21:16:15.356259 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:15.356227    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-bundle" (OuterVolumeSpecName: "bundle") pod "604b4550-7ea6-4b6d-aa30-767cf0b89a27" (UID: "604b4550-7ea6-4b6d-aa30-767cf0b89a27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:15.357042 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:15.357007    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604b4550-7ea6-4b6d-aa30-767cf0b89a27-kube-api-access-svrcl" (OuterVolumeSpecName: "kube-api-access-svrcl") pod "604b4550-7ea6-4b6d-aa30-767cf0b89a27" (UID: "604b4550-7ea6-4b6d-aa30-767cf0b89a27"). InnerVolumeSpecName "kube-api-access-svrcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:15.361415 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:15.361390    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-util" (OuterVolumeSpecName: "util") pod "604b4550-7ea6-4b6d-aa30-767cf0b89a27" (UID: "604b4550-7ea6-4b6d-aa30-767cf0b89a27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:15.455168 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:15.455132    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:16:15.455168 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:15.455170    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svrcl\" (UniqueName: \"kubernetes.io/projected/604b4550-7ea6-4b6d-aa30-767cf0b89a27-kube-api-access-svrcl\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:16:15.455394 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:15.455186    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/604b4550-7ea6-4b6d-aa30-767cf0b89a27-util\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:16:16.063312 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:16.063288    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x"
Sep 15 21:16:16.063479 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:16.063290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdbq7x" event={"ID":"604b4550-7ea6-4b6d-aa30-767cf0b89a27","Type":"ContainerDied","Data":"aceb176e65ef6d6185d96c59515f3ee947f1d0fc1697abf7af7b2ccbcae60cbb"}
Sep 15 21:16:16.063479 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:16.063402    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aceb176e65ef6d6185d96c59515f3ee947f1d0fc1697abf7af7b2ccbcae60cbb"
Sep 15 21:16:21.532313 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.532282    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"]
Sep 15 21:16:21.532744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.532604    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604b4550-7ea6-4b6d-aa30-767cf0b89a27" containerName="util"
Sep 15 21:16:21.532744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.532615    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="604b4550-7ea6-4b6d-aa30-767cf0b89a27" containerName="util"
Sep 15 21:16:21.532744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.532623    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604b4550-7ea6-4b6d-aa30-767cf0b89a27" containerName="extract"
Sep 15 21:16:21.532744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.532628    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="604b4550-7ea6-4b6d-aa30-767cf0b89a27" containerName="extract"
Sep 15 21:16:21.532744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.532639    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604b4550-7ea6-4b6d-aa30-767cf0b89a27" containerName="pull"
Sep 15 21:16:21.532744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.532644    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="604b4550-7ea6-4b6d-aa30-767cf0b89a27" containerName="pull"
Sep 15 21:16:21.532744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.532698    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="604b4550-7ea6-4b6d-aa30-767cf0b89a27" containerName="extract"
Sep 15 21:16:21.582071 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.582046    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:21.582202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.582051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"]
Sep 15 21:16:21.584255 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.584234    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 21:16:21.584402 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.584301    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 21:16:21.584986 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.584968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kjpxg\""
Sep 15 21:16:21.699764 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.699738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:21.699872 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.699769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2tfp\" (UniqueName: \"kubernetes.io/projected/6f332a29-7122-4b0a-8cff-a97d3e120437-kube-api-access-g2tfp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:21.699872 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.699790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:21.800571 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.800503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:21.800571 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.800534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2tfp\" (UniqueName: \"kubernetes.io/projected/6f332a29-7122-4b0a-8cff-a97d3e120437-kube-api-access-g2tfp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:21.800571 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.800558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:21.800938 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.800919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:21.800977 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.800928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:21.811070 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.811038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2tfp\" (UniqueName: \"kubernetes.io/projected/6f332a29-7122-4b0a-8cff-a97d3e120437-kube-api-access-g2tfp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:21.891416 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:21.891396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:22.162154 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:22.162132    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"]
Sep 15 21:16:22.163753 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:16:22.163727    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f332a29_7122_4b0a_8cff_a97d3e120437.slice/crio-848df12c3ca3d0d3cd8b86fa5644fd0eecbb686838df517599d93862d252144a WatchSource:0}: Error finding container 848df12c3ca3d0d3cd8b86fa5644fd0eecbb686838df517599d93862d252144a: Status 404 returned error can't find the container with id 848df12c3ca3d0d3cd8b86fa5644fd0eecbb686838df517599d93862d252144a
Sep 15 21:16:23.096649 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.096566    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f332a29-7122-4b0a-8cff-a97d3e120437" containerID="cdce28f45acde136e18fb5aa68a7779211cc1f776208b955a53d2eee95a0532e" exitCode=0
Sep 15 21:16:23.096649 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.096613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw" event={"ID":"6f332a29-7122-4b0a-8cff-a97d3e120437","Type":"ContainerDied","Data":"cdce28f45acde136e18fb5aa68a7779211cc1f776208b955a53d2eee95a0532e"}
Sep 15 21:16:23.096649 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.096645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw" event={"ID":"6f332a29-7122-4b0a-8cff-a97d3e120437","Type":"ContainerStarted","Data":"848df12c3ca3d0d3cd8b86fa5644fd0eecbb686838df517599d93862d252144a"}
Sep 15 21:16:23.882077 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.882052    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"]
Sep 15 21:16:23.884395 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.884380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:23.888766 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.888749    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 21:16:23.889878 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.889858    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 21:16:23.889968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.889901    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sp8bs\""
Sep 15 21:16:23.890039 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.890021    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 21:16:23.891398 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.891374    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 21:16:23.904520 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:23.904498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"]
Sep 15 21:16:24.020095 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.020074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdbd334c-230c-4d9b-8c8e-659d75336226-tmp\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.020204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.020169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdbd334c-230c-4d9b-8c8e-659d75336226-webhook-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.020270 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.020205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fdbd334c-230c-4d9b-8c8e-659d75336226-manifests\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.020270 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.020235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dffj8\" (UniqueName: \"kubernetes.io/projected/fdbd334c-230c-4d9b-8c8e-659d75336226-kube-api-access-dffj8\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.020367 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.020289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fdbd334c-230c-4d9b-8c8e-659d75336226-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.103398 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.103323    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f332a29-7122-4b0a-8cff-a97d3e120437" containerID="3c7b28ec1e44c42fa343791169de30ac3a2d27f88b52a8affc37353122d1779f" exitCode=0
Sep 15 21:16:24.103740 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.103408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw" event={"ID":"6f332a29-7122-4b0a-8cff-a97d3e120437","Type":"ContainerDied","Data":"3c7b28ec1e44c42fa343791169de30ac3a2d27f88b52a8affc37353122d1779f"}
Sep 15 21:16:24.120593 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.120573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdbd334c-230c-4d9b-8c8e-659d75336226-webhook-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.120688 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.120602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fdbd334c-230c-4d9b-8c8e-659d75336226-manifests\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.120688 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.120620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dffj8\" (UniqueName: \"kubernetes.io/projected/fdbd334c-230c-4d9b-8c8e-659d75336226-kube-api-access-dffj8\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.120688 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.120666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fdbd334c-230c-4d9b-8c8e-659d75336226-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.120844 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.120701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdbd334c-230c-4d9b-8c8e-659d75336226-tmp\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.120995 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.120965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fdbd334c-230c-4d9b-8c8e-659d75336226-manifests\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.123031 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.123009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdbd334c-230c-4d9b-8c8e-659d75336226-tmp\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.123124 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.123096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdbd334c-230c-4d9b-8c8e-659d75336226-webhook-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.123391 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.123374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fdbd334c-230c-4d9b-8c8e-659d75336226-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.128621 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.128598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dffj8\" (UniqueName: \"kubernetes.io/projected/fdbd334c-230c-4d9b-8c8e-659d75336226-kube-api-access-dffj8\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-7s6pt\" (UID: \"fdbd334c-230c-4d9b-8c8e-659d75336226\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.201977 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.201954    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:24.476110 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:24.476083    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"]
Sep 15 21:16:24.478254 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:16:24.478228    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbd334c_230c_4d9b_8c8e_659d75336226.slice/crio-c24440aff0b28839924fb93c4f1642f4277553e5a4021ccdb663d9861d050ad8 WatchSource:0}: Error finding container c24440aff0b28839924fb93c4f1642f4277553e5a4021ccdb663d9861d050ad8: Status 404 returned error can't find the container with id c24440aff0b28839924fb93c4f1642f4277553e5a4021ccdb663d9861d050ad8
Sep 15 21:16:25.111787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:25.111703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt" event={"ID":"fdbd334c-230c-4d9b-8c8e-659d75336226","Type":"ContainerStarted","Data":"c24440aff0b28839924fb93c4f1642f4277553e5a4021ccdb663d9861d050ad8"}
Sep 15 21:16:25.114466 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:25.114438    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f332a29-7122-4b0a-8cff-a97d3e120437" containerID="75b356a356ef90612da10e5e6f6c44e36ab617ee0ae368d48ff7a23a390a9c8e" exitCode=0
Sep 15 21:16:25.114578 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:25.114485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw" event={"ID":"6f332a29-7122-4b0a-8cff-a97d3e120437","Type":"ContainerDied","Data":"75b356a356ef90612da10e5e6f6c44e36ab617ee0ae368d48ff7a23a390a9c8e"}
Sep 15 21:16:27.420645 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:27.420624    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:27.449879 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:27.449854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-bundle\") pod \"6f332a29-7122-4b0a-8cff-a97d3e120437\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") "
Sep 15 21:16:27.449982 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:27.449893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-util\") pod \"6f332a29-7122-4b0a-8cff-a97d3e120437\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") "
Sep 15 21:16:27.449982 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:27.449922    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2tfp\" (UniqueName: \"kubernetes.io/projected/6f332a29-7122-4b0a-8cff-a97d3e120437-kube-api-access-g2tfp\") pod \"6f332a29-7122-4b0a-8cff-a97d3e120437\" (UID: \"6f332a29-7122-4b0a-8cff-a97d3e120437\") "
Sep 15 21:16:27.450748 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:27.450718    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-bundle" (OuterVolumeSpecName: "bundle") pod "6f332a29-7122-4b0a-8cff-a97d3e120437" (UID: "6f332a29-7122-4b0a-8cff-a97d3e120437"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:27.452240 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:27.452215    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f332a29-7122-4b0a-8cff-a97d3e120437-kube-api-access-g2tfp" (OuterVolumeSpecName: "kube-api-access-g2tfp") pod "6f332a29-7122-4b0a-8cff-a97d3e120437" (UID: "6f332a29-7122-4b0a-8cff-a97d3e120437"). InnerVolumeSpecName "kube-api-access-g2tfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:27.457260 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:27.457218    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-util" (OuterVolumeSpecName: "util") pod "6f332a29-7122-4b0a-8cff-a97d3e120437" (UID: "6f332a29-7122-4b0a-8cff-a97d3e120437"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:16:27.551153 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:27.551127    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2tfp\" (UniqueName: \"kubernetes.io/projected/6f332a29-7122-4b0a-8cff-a97d3e120437-kube-api-access-g2tfp\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:16:27.551153 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:27.551152    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:16:27.551283 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:27.551161    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f332a29-7122-4b0a-8cff-a97d3e120437-util\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:16:28.134940 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:28.134911    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw"
Sep 15 21:16:28.135103 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:28.134942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02dfdw" event={"ID":"6f332a29-7122-4b0a-8cff-a97d3e120437","Type":"ContainerDied","Data":"848df12c3ca3d0d3cd8b86fa5644fd0eecbb686838df517599d93862d252144a"}
Sep 15 21:16:28.135103 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:28.134989    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="848df12c3ca3d0d3cd8b86fa5644fd0eecbb686838df517599d93862d252144a"
Sep 15 21:16:28.136716 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:28.136684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt" event={"ID":"fdbd334c-230c-4d9b-8c8e-659d75336226","Type":"ContainerStarted","Data":"9789d0d4723267dee2480a4abcb453c42bb6dfc3e822144f13405ba7ec405fcf"}
Sep 15 21:16:29.142772 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:29.142741    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdbd334c-230c-4d9b-8c8e-659d75336226" containerID="9789d0d4723267dee2480a4abcb453c42bb6dfc3e822144f13405ba7ec405fcf" exitCode=0
Sep 15 21:16:29.143107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:29.142795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt" event={"ID":"fdbd334c-230c-4d9b-8c8e-659d75336226","Type":"ContainerDied","Data":"9789d0d4723267dee2480a4abcb453c42bb6dfc3e822144f13405ba7ec405fcf"}
Sep 15 21:16:30.149189 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:30.149158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt" event={"ID":"fdbd334c-230c-4d9b-8c8e-659d75336226","Type":"ContainerStarted","Data":"54f6a13f3cb8184f4ddf3ee6c224219c13226c43a6eb93aab0248e5501c4bc3f"}
Sep 15 21:16:30.149602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:30.149274    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:30.172055 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:30.172013    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt" podStartSLOduration=2.886794305 podStartE2EDuration="7.171999431s" totalImagesPullingTime="3.263339341s" totalInitContainerRuntime="1.021865785s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:30.170465414 +0000 UTC m=+380.133522797" watchObservedRunningTime="2026-09-15 21:16:30.171999431 +0000 UTC m=+380.135056811"
Sep 15 21:16:41.156748 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.156721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-7s6pt"
Sep 15 21:16:41.268263 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.268232    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"]
Sep 15 21:16:41.268678 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.268620    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f332a29-7122-4b0a-8cff-a97d3e120437" containerName="extract"
Sep 15 21:16:41.268678 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.268676    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f332a29-7122-4b0a-8cff-a97d3e120437" containerName="extract"
Sep 15 21:16:41.268883 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.268685    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f332a29-7122-4b0a-8cff-a97d3e120437" containerName="util"
Sep 15 21:16:41.268883 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.268693    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f332a29-7122-4b0a-8cff-a97d3e120437" containerName="util"
Sep 15 21:16:41.268883 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.268702    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f332a29-7122-4b0a-8cff-a97d3e120437" containerName="pull"
Sep 15 21:16:41.268883 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.268709    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f332a29-7122-4b0a-8cff-a97d3e120437" containerName="pull"
Sep 15 21:16:41.268883 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.268798    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f332a29-7122-4b0a-8cff-a97d3e120437" containerName="extract"
Sep 15 21:16:41.272774 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.272757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.275027 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.275011    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 21:16:41.275027 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.275019    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 21:16:41.275195 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.275016    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q2mw2\""
Sep 15 21:16:41.275195 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.275100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 21:16:41.280954 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.280934    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"]
Sep 15 21:16:41.357368 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.357323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b86bb871-3883-4b9c-b98c-bafa5963bdd6-metrics-cert\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.357507 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.357379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b86bb871-3883-4b9c-b98c-bafa5963bdd6-cert\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.357507 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.357488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvdb\" (UniqueName: \"kubernetes.io/projected/b86bb871-3883-4b9c-b98c-bafa5963bdd6-kube-api-access-shvdb\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.357596 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.357513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b86bb871-3883-4b9c-b98c-bafa5963bdd6-manager-config\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.458483 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.458406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvdb\" (UniqueName: \"kubernetes.io/projected/b86bb871-3883-4b9c-b98c-bafa5963bdd6-kube-api-access-shvdb\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.458483 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.458446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b86bb871-3883-4b9c-b98c-bafa5963bdd6-manager-config\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.458483 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.458488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b86bb871-3883-4b9c-b98c-bafa5963bdd6-metrics-cert\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.458717 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.458505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b86bb871-3883-4b9c-b98c-bafa5963bdd6-cert\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.459191 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.459164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b86bb871-3883-4b9c-b98c-bafa5963bdd6-manager-config\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.460901 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.460881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b86bb871-3883-4b9c-b98c-bafa5963bdd6-cert\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.460990 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.460937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b86bb871-3883-4b9c-b98c-bafa5963bdd6-metrics-cert\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.467936 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.467911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvdb\" (UniqueName: \"kubernetes.io/projected/b86bb871-3883-4b9c-b98c-bafa5963bdd6-kube-api-access-shvdb\") pod \"lws-controller-manager-ffdbb6cbd-h7f66\" (UID: \"b86bb871-3883-4b9c-b98c-bafa5963bdd6\") " pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.582714 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.582694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:41.852403 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:41.852381    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"]
Sep 15 21:16:41.855154 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:16:41.855133    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86bb871_3883_4b9c_b98c_bafa5963bdd6.slice/crio-d9da4540973ed3cc08df0b023a8a30df33cab620acaebb1e84c729ab916b1bd4 WatchSource:0}: Error finding container d9da4540973ed3cc08df0b023a8a30df33cab620acaebb1e84c729ab916b1bd4: Status 404 returned error can't find the container with id d9da4540973ed3cc08df0b023a8a30df33cab620acaebb1e84c729ab916b1bd4
Sep 15 21:16:42.211202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:42.211168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66" event={"ID":"b86bb871-3883-4b9c-b98c-bafa5963bdd6","Type":"ContainerStarted","Data":"d9da4540973ed3cc08df0b023a8a30df33cab620acaebb1e84c729ab916b1bd4"}
Sep 15 21:16:44.225034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:44.224997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66" event={"ID":"b86bb871-3883-4b9c-b98c-bafa5963bdd6","Type":"ContainerStarted","Data":"73fbf2723bbced45ba74a6e784f173bd19755b1ec34d46f30896913ffd5555ea"}
Sep 15 21:16:44.225435 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:44.225134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:16:44.244324 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:44.244280    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66" podStartSLOduration=1.438005874 podStartE2EDuration="3.244265095s" totalImagesPullingTime="1.806259221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:44.242965398 +0000 UTC m=+394.206022774" watchObservedRunningTime="2026-09-15 21:16:44.244265095 +0000 UTC m=+394.207322476"
Sep 15 21:16:49.276466 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.276430    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"]
Sep 15 21:16:49.279894 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.279878    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.282035 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.282009    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 21:16:49.282154 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.282134    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 21:16:49.282755 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.282718    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 21:16:49.282755 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.282752    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rfh6w\""
Sep 15 21:16:49.282918 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.282793    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 21:16:49.290091 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.290070    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"]
Sep 15 21:16:49.327708 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.327672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642lz\" (UniqueName: \"kubernetes.io/projected/260f8851-3890-494c-99a4-0062bbc2327c-kube-api-access-642lz\") pod \"kube-auth-proxy-cbb5b4bf7-8wvnx\" (UID: \"260f8851-3890-494c-99a4-0062bbc2327c\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.327889 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.327713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/260f8851-3890-494c-99a4-0062bbc2327c-tmp\") pod \"kube-auth-proxy-cbb5b4bf7-8wvnx\" (UID: \"260f8851-3890-494c-99a4-0062bbc2327c\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.327889 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.327756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260f8851-3890-494c-99a4-0062bbc2327c-tls-certs\") pod \"kube-auth-proxy-cbb5b4bf7-8wvnx\" (UID: \"260f8851-3890-494c-99a4-0062bbc2327c\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.428594 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.428559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642lz\" (UniqueName: \"kubernetes.io/projected/260f8851-3890-494c-99a4-0062bbc2327c-kube-api-access-642lz\") pod \"kube-auth-proxy-cbb5b4bf7-8wvnx\" (UID: \"260f8851-3890-494c-99a4-0062bbc2327c\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.428594 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.428596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/260f8851-3890-494c-99a4-0062bbc2327c-tmp\") pod \"kube-auth-proxy-cbb5b4bf7-8wvnx\" (UID: \"260f8851-3890-494c-99a4-0062bbc2327c\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.428835 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.428622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260f8851-3890-494c-99a4-0062bbc2327c-tls-certs\") pod \"kube-auth-proxy-cbb5b4bf7-8wvnx\" (UID: \"260f8851-3890-494c-99a4-0062bbc2327c\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.430868 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.430839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/260f8851-3890-494c-99a4-0062bbc2327c-tmp\") pod \"kube-auth-proxy-cbb5b4bf7-8wvnx\" (UID: \"260f8851-3890-494c-99a4-0062bbc2327c\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.431018 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.430998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260f8851-3890-494c-99a4-0062bbc2327c-tls-certs\") pod \"kube-auth-proxy-cbb5b4bf7-8wvnx\" (UID: \"260f8851-3890-494c-99a4-0062bbc2327c\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.436864 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.436843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-642lz\" (UniqueName: \"kubernetes.io/projected/260f8851-3890-494c-99a4-0062bbc2327c-kube-api-access-642lz\") pod \"kube-auth-proxy-cbb5b4bf7-8wvnx\" (UID: \"260f8851-3890-494c-99a4-0062bbc2327c\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.591529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.591450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"
Sep 15 21:16:49.859920 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:49.859887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx"]
Sep 15 21:16:49.862035 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:16:49.861994    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260f8851_3890_494c_99a4_0062bbc2327c.slice/crio-3e5498aafe4ae562c7ab0289d12ce93058a3a7d6350d7a416ae66e1433747b37 WatchSource:0}: Error finding container 3e5498aafe4ae562c7ab0289d12ce93058a3a7d6350d7a416ae66e1433747b37: Status 404 returned error can't find the container with id 3e5498aafe4ae562c7ab0289d12ce93058a3a7d6350d7a416ae66e1433747b37
Sep 15 21:16:50.257235 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:50.257201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx" event={"ID":"260f8851-3890-494c-99a4-0062bbc2327c","Type":"ContainerStarted","Data":"3e5498aafe4ae562c7ab0289d12ce93058a3a7d6350d7a416ae66e1433747b37"}
Sep 15 21:16:53.274507 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:53.274472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx" event={"ID":"260f8851-3890-494c-99a4-0062bbc2327c","Type":"ContainerStarted","Data":"29343521114b8a9baeea6bffb9fa0f15e00da883b200cb6b495e03e29a53c2da"}
Sep 15 21:16:53.294079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:53.294025    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-8wvnx" podStartSLOduration=1.885972872 podStartE2EDuration="4.294006488s" totalImagesPullingTime="2.408033616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:53.292187234 +0000 UTC m=+403.255244615" watchObservedRunningTime="2026-09-15 21:16:53.294006488 +0000 UTC m=+403.257063869"
Sep 15 21:16:55.232976 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:16:55.232948    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-ffdbb6cbd-h7f66"
Sep 15 21:17:59.664118 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.664077    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"]
Sep 15 21:17:59.666933 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.666917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:17:59.669163 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.669140    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 21:17:59.669294 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.669141    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 21:17:59.669833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.669810    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kjpxg\""
Sep 15 21:17:59.675912 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.675891    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"]
Sep 15 21:17:59.748917 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.748880    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"]
Sep 15 21:17:59.751721 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.751699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:17:59.760307 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.760284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"]
Sep 15 21:17:59.837856 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.837822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:17:59.838042 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.837863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:17:59.838042 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.837950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqss\" (UniqueName: \"kubernetes.io/projected/25503b39-e3ef-4d11-b129-b9e8301e8b29-kube-api-access-pxqss\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:17:59.838042 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.838021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h45s\" (UniqueName: \"kubernetes.io/projected/cf5de0c5-9beb-423a-8829-1a836e83c5c1-kube-api-access-5h45s\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:17:59.838193 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.838048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:17:59.838193 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.838103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:17:59.853775 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.853743    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"]
Sep 15 21:17:59.856481 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.856466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:17:59.866212 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.866188    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"]
Sep 15 21:17:59.938728 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.938642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h45s\" (UniqueName: \"kubernetes.io/projected/cf5de0c5-9beb-423a-8829-1a836e83c5c1-kube-api-access-5h45s\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:17:59.938728 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.938680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:17:59.938728 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.938701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjcqx\" (UniqueName: \"kubernetes.io/projected/9e4bee24-24f7-4e15-9ede-694573ea93d3-kube-api-access-cjcqx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:17:59.938728 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.938724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:17:59.938968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.938866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:17:59.938968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.938920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:17:59.938968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.938946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:17:59.939110 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.939019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:17:59.939110 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.939047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:17:59.939110 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.939055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqss\" (UniqueName: \"kubernetes.io/projected/25503b39-e3ef-4d11-b129-b9e8301e8b29-kube-api-access-pxqss\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:17:59.939301 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.939283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:17:59.939376 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.939312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:17:59.939376 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.939336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:17:59.949033 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.949007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqss\" (UniqueName: \"kubernetes.io/projected/25503b39-e3ef-4d11-b129-b9e8301e8b29-kube-api-access-pxqss\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:17:59.949392 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.949371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h45s\" (UniqueName: \"kubernetes.io/projected/cf5de0c5-9beb-423a-8829-1a836e83c5c1-kube-api-access-5h45s\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:17:59.953330 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.953305    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"]
Sep 15 21:17:59.956091 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.956074    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:17:59.964682 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.964655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"]
Sep 15 21:17:59.977456 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:17:59.977431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:18:00.040152 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.040065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:18:00.040325 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.040150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:00.040325 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.040198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:00.040325 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.040228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjcqx\" (UniqueName: \"kubernetes.io/projected/9e4bee24-24f7-4e15-9ede-694573ea93d3-kube-api-access-cjcqx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:18:00.040325 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.040265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:18:00.040325 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.040302    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wff6\" (UniqueName: \"kubernetes.io/projected/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-kube-api-access-5wff6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:00.041083 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.040764    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:18:00.041083 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.041038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:18:00.050937 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.050910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjcqx\" (UniqueName: \"kubernetes.io/projected/9e4bee24-24f7-4e15-9ede-694573ea93d3-kube-api-access-cjcqx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:18:00.061620 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.061584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:18:00.141016 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.140980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wff6\" (UniqueName: \"kubernetes.io/projected/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-kube-api-access-5wff6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:00.141871 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.141443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:00.141871 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.141530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:00.142071 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.142001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:00.142071 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.142056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:00.151298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.151267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wff6\" (UniqueName: \"kubernetes.io/projected/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-kube-api-access-5wff6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:00.166008 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.165971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:18:00.248417 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.248395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"]
Sep 15 21:18:00.250953 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:18:00.250905    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25503b39_e3ef_4d11_b129_b9e8301e8b29.slice/crio-91f8f4e7d3fd547f2a9990267be5cdf8d98862bc91b5417077f4706669a18613 WatchSource:0}: Error finding container 91f8f4e7d3fd547f2a9990267be5cdf8d98862bc91b5417077f4706669a18613: Status 404 returned error can't find the container with id 91f8f4e7d3fd547f2a9990267be5cdf8d98862bc91b5417077f4706669a18613
Sep 15 21:18:00.275022 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.274997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:00.341212 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.341126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"]
Sep 15 21:18:00.440782 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.440754    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"]
Sep 15 21:18:00.441720 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:18:00.441695    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4bee24_24f7_4e15_9ede_694573ea93d3.slice/crio-2258d68273570ef6198d09f12db11fad01d310237805356452e0116c890b625f WatchSource:0}: Error finding container 2258d68273570ef6198d09f12db11fad01d310237805356452e0116c890b625f: Status 404 returned error can't find the container with id 2258d68273570ef6198d09f12db11fad01d310237805356452e0116c890b625f
Sep 15 21:18:00.562956 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.562929    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"]
Sep 15 21:18:00.591212 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:18:00.591173    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb0bd64_16a6_47e4_a8ea_5630517aa2a1.slice/crio-4ec5651cc498c337cf4589bb00dc417e0ca2ddeed75fbc749077c03eb77569c7 WatchSource:0}: Error finding container 4ec5651cc498c337cf4589bb00dc417e0ca2ddeed75fbc749077c03eb77569c7: Status 404 returned error can't find the container with id 4ec5651cc498c337cf4589bb00dc417e0ca2ddeed75fbc749077c03eb77569c7
Sep 15 21:18:00.607551 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.607517    2563 generic.go:356] "Generic (PLEG): container finished" podID="9e4bee24-24f7-4e15-9ede-694573ea93d3" containerID="092ea21f34d11cf9a32c368b43438aaf77acca645b0c621e21aa00223cf856e1" exitCode=0
Sep 15 21:18:00.607666 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.607596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w" event={"ID":"9e4bee24-24f7-4e15-9ede-694573ea93d3","Type":"ContainerDied","Data":"092ea21f34d11cf9a32c368b43438aaf77acca645b0c621e21aa00223cf856e1"}
Sep 15 21:18:00.607666 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.607635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w" event={"ID":"9e4bee24-24f7-4e15-9ede-694573ea93d3","Type":"ContainerStarted","Data":"2258d68273570ef6198d09f12db11fad01d310237805356452e0116c890b625f"}
Sep 15 21:18:00.609587 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.609560    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf5de0c5-9beb-423a-8829-1a836e83c5c1" containerID="85c3834a64330fc55b89b1aeef2682f0646bac470d53c07a074a35926c8a009b" exitCode=0
Sep 15 21:18:00.609840 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.609602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c" event={"ID":"cf5de0c5-9beb-423a-8829-1a836e83c5c1","Type":"ContainerDied","Data":"85c3834a64330fc55b89b1aeef2682f0646bac470d53c07a074a35926c8a009b"}
Sep 15 21:18:00.609959 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.609875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c" event={"ID":"cf5de0c5-9beb-423a-8829-1a836e83c5c1","Type":"ContainerStarted","Data":"92978f5d3bc2ef9606b99d1a94b748a257bce429ad26b1d44077b5f7a1c178f5"}
Sep 15 21:18:00.612095 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.612063    2563 generic.go:356] "Generic (PLEG): container finished" podID="25503b39-e3ef-4d11-b129-b9e8301e8b29" containerID="4bba6302b2def079cfe2e66dfe51f42314ca45aac88ec0177dd56673106b5c4f" exitCode=0
Sep 15 21:18:00.612201 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.612153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s" event={"ID":"25503b39-e3ef-4d11-b129-b9e8301e8b29","Type":"ContainerDied","Data":"4bba6302b2def079cfe2e66dfe51f42314ca45aac88ec0177dd56673106b5c4f"}
Sep 15 21:18:00.612257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.612210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s" event={"ID":"25503b39-e3ef-4d11-b129-b9e8301e8b29","Type":"ContainerStarted","Data":"91f8f4e7d3fd547f2a9990267be5cdf8d98862bc91b5417077f4706669a18613"}
Sep 15 21:18:00.614219 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:00.614192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl" event={"ID":"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1","Type":"ContainerStarted","Data":"4ec5651cc498c337cf4589bb00dc417e0ca2ddeed75fbc749077c03eb77569c7"}
Sep 15 21:18:01.621582 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:01.621545    2563 generic.go:356] "Generic (PLEG): container finished" podID="25503b39-e3ef-4d11-b129-b9e8301e8b29" containerID="d3d9f41c310924aa071d40467a677110bf526941ef11ad37e77b517000b9c408" exitCode=0
Sep 15 21:18:01.622002 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:01.621622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s" event={"ID":"25503b39-e3ef-4d11-b129-b9e8301e8b29","Type":"ContainerDied","Data":"d3d9f41c310924aa071d40467a677110bf526941ef11ad37e77b517000b9c408"}
Sep 15 21:18:01.623309 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:01.623283    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" containerID="e0274fa0e1f8deb4ab54ccd2f44517f539ae6c95089929a8beaff537fa5ff66e" exitCode=0
Sep 15 21:18:01.623433 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:01.623382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl" event={"ID":"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1","Type":"ContainerDied","Data":"e0274fa0e1f8deb4ab54ccd2f44517f539ae6c95089929a8beaff537fa5ff66e"}
Sep 15 21:18:02.630742 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:02.630711    2563 generic.go:356] "Generic (PLEG): container finished" podID="9e4bee24-24f7-4e15-9ede-694573ea93d3" containerID="9369973e9ab48fab1cf78acebf9c13a7bcf8c78bc23c7f87c3b5317c871b617d" exitCode=0
Sep 15 21:18:02.631172 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:02.630783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w" event={"ID":"9e4bee24-24f7-4e15-9ede-694573ea93d3","Type":"ContainerDied","Data":"9369973e9ab48fab1cf78acebf9c13a7bcf8c78bc23c7f87c3b5317c871b617d"}
Sep 15 21:18:02.632846 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:02.632816    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf5de0c5-9beb-423a-8829-1a836e83c5c1" containerID="a950f056556d50a6ed463b666adbf2166c21d1d0d3eee77615f3da801aa29ab6" exitCode=0
Sep 15 21:18:02.632941 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:02.632886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c" event={"ID":"cf5de0c5-9beb-423a-8829-1a836e83c5c1","Type":"ContainerDied","Data":"a950f056556d50a6ed463b666adbf2166c21d1d0d3eee77615f3da801aa29ab6"}
Sep 15 21:18:02.635455 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:02.635433    2563 generic.go:356] "Generic (PLEG): container finished" podID="25503b39-e3ef-4d11-b129-b9e8301e8b29" containerID="728ce2050773ea3184b9d0ecbd57b1a1794aed0bc8106c985c50a8b57d6cf43a" exitCode=0
Sep 15 21:18:02.635523 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:02.635477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s" event={"ID":"25503b39-e3ef-4d11-b129-b9e8301e8b29","Type":"ContainerDied","Data":"728ce2050773ea3184b9d0ecbd57b1a1794aed0bc8106c985c50a8b57d6cf43a"}
Sep 15 21:18:03.408213 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.408178    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-587c544fd5-x9xp7"]
Sep 15 21:18:03.410754 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.410737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.421242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.421218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-587c544fd5-x9xp7"]
Sep 15 21:18:03.575266 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.575228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-console-config\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.575467 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.575279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-service-ca\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.575467 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.575385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-trusted-ca-bundle\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.575467 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.575452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6481e213-1f4a-4933-bf91-ce500d6ae171-console-serving-cert\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.575573 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.575486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhjf\" (UniqueName: \"kubernetes.io/projected/6481e213-1f4a-4933-bf91-ce500d6ae171-kube-api-access-6nhjf\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.575573 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.575532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6481e213-1f4a-4933-bf91-ce500d6ae171-console-oauth-config\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.575573 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.575552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-oauth-serving-cert\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.642724 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.642689    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" containerID="37d23719c8589d5e3d47acf450fd111a15d5933c3b6006adfe79cb706ef2017f" exitCode=0
Sep 15 21:18:03.643139 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.642765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl" event={"ID":"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1","Type":"ContainerDied","Data":"37d23719c8589d5e3d47acf450fd111a15d5933c3b6006adfe79cb706ef2017f"}
Sep 15 21:18:03.645035 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.645009    2563 generic.go:356] "Generic (PLEG): container finished" podID="9e4bee24-24f7-4e15-9ede-694573ea93d3" containerID="8d7f18d4a6227348f75cc33d9fe81f8aca475848572bc3ba57999026275d508a" exitCode=0
Sep 15 21:18:03.645134 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.645064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w" event={"ID":"9e4bee24-24f7-4e15-9ede-694573ea93d3","Type":"ContainerDied","Data":"8d7f18d4a6227348f75cc33d9fe81f8aca475848572bc3ba57999026275d508a"}
Sep 15 21:18:03.647281 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.647262    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf5de0c5-9beb-423a-8829-1a836e83c5c1" containerID="a6883230db3fd01c771ac2300dd907170a8998047b16e5061b7a03d33f4b5f68" exitCode=0
Sep 15 21:18:03.647390 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.647329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c" event={"ID":"cf5de0c5-9beb-423a-8829-1a836e83c5c1","Type":"ContainerDied","Data":"a6883230db3fd01c771ac2300dd907170a8998047b16e5061b7a03d33f4b5f68"}
Sep 15 21:18:03.677569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.676958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-console-config\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.677569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.677033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-service-ca\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.677569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.677077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-trusted-ca-bundle\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.677569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.677122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6481e213-1f4a-4933-bf91-ce500d6ae171-console-serving-cert\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.677569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.677162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhjf\" (UniqueName: \"kubernetes.io/projected/6481e213-1f4a-4933-bf91-ce500d6ae171-kube-api-access-6nhjf\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.677569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.677214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6481e213-1f4a-4933-bf91-ce500d6ae171-console-oauth-config\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.677569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.677237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-oauth-serving-cert\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.678131 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.678106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-oauth-serving-cert\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.678420 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.678257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-trusted-ca-bundle\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.679090 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.678860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-console-config\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.679238 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.679167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6481e213-1f4a-4933-bf91-ce500d6ae171-service-ca\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.681479 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.681455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6481e213-1f4a-4933-bf91-ce500d6ae171-console-serving-cert\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.682053 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.682027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6481e213-1f4a-4933-bf91-ce500d6ae171-console-oauth-config\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.687412 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.687308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhjf\" (UniqueName: \"kubernetes.io/projected/6481e213-1f4a-4933-bf91-ce500d6ae171-kube-api-access-6nhjf\") pod \"console-587c544fd5-x9xp7\" (UID: \"6481e213-1f4a-4933-bf91-ce500d6ae171\") " pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.720962 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.720930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:03.779074 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.778890    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:18:03.879429 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.879393    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-bundle\") pod \"25503b39-e3ef-4d11-b129-b9e8301e8b29\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") "
Sep 15 21:18:03.879591 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.879479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-util\") pod \"25503b39-e3ef-4d11-b129-b9e8301e8b29\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") "
Sep 15 21:18:03.879591 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.879510    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxqss\" (UniqueName: \"kubernetes.io/projected/25503b39-e3ef-4d11-b129-b9e8301e8b29-kube-api-access-pxqss\") pod \"25503b39-e3ef-4d11-b129-b9e8301e8b29\" (UID: \"25503b39-e3ef-4d11-b129-b9e8301e8b29\") "
Sep 15 21:18:03.880032 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.880006    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-bundle" (OuterVolumeSpecName: "bundle") pod "25503b39-e3ef-4d11-b129-b9e8301e8b29" (UID: "25503b39-e3ef-4d11-b129-b9e8301e8b29"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:03.881601 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.881581    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25503b39-e3ef-4d11-b129-b9e8301e8b29-kube-api-access-pxqss" (OuterVolumeSpecName: "kube-api-access-pxqss") pod "25503b39-e3ef-4d11-b129-b9e8301e8b29" (UID: "25503b39-e3ef-4d11-b129-b9e8301e8b29"). InnerVolumeSpecName "kube-api-access-pxqss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:03.884410 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.884386    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-util" (OuterVolumeSpecName: "util") pod "25503b39-e3ef-4d11-b129-b9e8301e8b29" (UID: "25503b39-e3ef-4d11-b129-b9e8301e8b29"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:03.980274 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.980240    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-util\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:03.980274 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.980270    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxqss\" (UniqueName: \"kubernetes.io/projected/25503b39-e3ef-4d11-b129-b9e8301e8b29-kube-api-access-pxqss\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:03.980274 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.980280    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25503b39-e3ef-4d11-b129-b9e8301e8b29-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:03.999747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:03.999716    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-587c544fd5-x9xp7"]
Sep 15 21:18:04.001146 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:18:04.001122    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6481e213_1f4a_4933_bf91_ce500d6ae171.slice/crio-85c3b63a355458812b0657cd5343941edf003f4f06b30fd12588b036620bfa6a WatchSource:0}: Error finding container 85c3b63a355458812b0657cd5343941edf003f4f06b30fd12588b036620bfa6a: Status 404 returned error can't find the container with id 85c3b63a355458812b0657cd5343941edf003f4f06b30fd12588b036620bfa6a
Sep 15 21:18:04.658793 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.658758    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" containerID="99975497ed59f4afa6acda9a50b94482cd7a2609063c09a480f52120e96814b0" exitCode=0
Sep 15 21:18:04.659246 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.658840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl" event={"ID":"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1","Type":"ContainerDied","Data":"99975497ed59f4afa6acda9a50b94482cd7a2609063c09a480f52120e96814b0"}
Sep 15 21:18:04.660486 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.660459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587c544fd5-x9xp7" event={"ID":"6481e213-1f4a-4933-bf91-ce500d6ae171","Type":"ContainerStarted","Data":"ac783558d84e0c1e347328376d5189cadff37f8a4e854973a9811b9a754925b2"}
Sep 15 21:18:04.660607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.660499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587c544fd5-x9xp7" event={"ID":"6481e213-1f4a-4933-bf91-ce500d6ae171","Type":"ContainerStarted","Data":"85c3b63a355458812b0657cd5343941edf003f4f06b30fd12588b036620bfa6a"}
Sep 15 21:18:04.662250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.662232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s" event={"ID":"25503b39-e3ef-4d11-b129-b9e8301e8b29","Type":"ContainerDied","Data":"91f8f4e7d3fd547f2a9990267be5cdf8d98862bc91b5417077f4706669a18613"}
Sep 15 21:18:04.662327 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.662263    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91f8f4e7d3fd547f2a9990267be5cdf8d98862bc91b5417077f4706669a18613"
Sep 15 21:18:04.662327 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.662261    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az8p5s"
Sep 15 21:18:04.700302 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.700245    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-587c544fd5-x9xp7" podStartSLOduration=1.7002252709999999 podStartE2EDuration="1.700225271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:04.698341513 +0000 UTC m=+474.661398892" watchObservedRunningTime="2026-09-15 21:18:04.700225271 +0000 UTC m=+474.663282652"
Sep 15 21:18:04.802819 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.802785    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:18:04.814043 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.814009    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:18:04.989430 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.989396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-bundle\") pod \"9e4bee24-24f7-4e15-9ede-694573ea93d3\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") "
Sep 15 21:18:04.989604 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.989485    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h45s\" (UniqueName: \"kubernetes.io/projected/cf5de0c5-9beb-423a-8829-1a836e83c5c1-kube-api-access-5h45s\") pod \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") "
Sep 15 21:18:04.989604 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.989532    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjcqx\" (UniqueName: \"kubernetes.io/projected/9e4bee24-24f7-4e15-9ede-694573ea93d3-kube-api-access-cjcqx\") pod \"9e4bee24-24f7-4e15-9ede-694573ea93d3\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") "
Sep 15 21:18:04.989604 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.989583    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-bundle\") pod \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") "
Sep 15 21:18:04.989787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.989624    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-util\") pod \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\" (UID: \"cf5de0c5-9beb-423a-8829-1a836e83c5c1\") "
Sep 15 21:18:04.989787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.989659    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-util\") pod \"9e4bee24-24f7-4e15-9ede-694573ea93d3\" (UID: \"9e4bee24-24f7-4e15-9ede-694573ea93d3\") "
Sep 15 21:18:04.990201 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.990168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-bundle" (OuterVolumeSpecName: "bundle") pod "9e4bee24-24f7-4e15-9ede-694573ea93d3" (UID: "9e4bee24-24f7-4e15-9ede-694573ea93d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:04.990307 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.990257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-bundle" (OuterVolumeSpecName: "bundle") pod "cf5de0c5-9beb-423a-8829-1a836e83c5c1" (UID: "cf5de0c5-9beb-423a-8829-1a836e83c5c1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:04.991991 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.991959    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5de0c5-9beb-423a-8829-1a836e83c5c1-kube-api-access-5h45s" (OuterVolumeSpecName: "kube-api-access-5h45s") pod "cf5de0c5-9beb-423a-8829-1a836e83c5c1" (UID: "cf5de0c5-9beb-423a-8829-1a836e83c5c1"). InnerVolumeSpecName "kube-api-access-5h45s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:04.992394 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.992336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4bee24-24f7-4e15-9ede-694573ea93d3-kube-api-access-cjcqx" (OuterVolumeSpecName: "kube-api-access-cjcqx") pod "9e4bee24-24f7-4e15-9ede-694573ea93d3" (UID: "9e4bee24-24f7-4e15-9ede-694573ea93d3"). InnerVolumeSpecName "kube-api-access-cjcqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:04.995062 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.995038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-util" (OuterVolumeSpecName: "util") pod "9e4bee24-24f7-4e15-9ede-694573ea93d3" (UID: "9e4bee24-24f7-4e15-9ede-694573ea93d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:04.996140 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:04.996098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-util" (OuterVolumeSpecName: "util") pod "cf5de0c5-9beb-423a-8829-1a836e83c5c1" (UID: "cf5de0c5-9beb-423a-8829-1a836e83c5c1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:05.091257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.091169    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:05.091257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.091198    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf5de0c5-9beb-423a-8829-1a836e83c5c1-util\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:05.091257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.091207    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-util\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:05.091257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.091215    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e4bee24-24f7-4e15-9ede-694573ea93d3-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:05.091257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.091224    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h45s\" (UniqueName: \"kubernetes.io/projected/cf5de0c5-9beb-423a-8829-1a836e83c5c1-kube-api-access-5h45s\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:05.091257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.091234    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjcqx\" (UniqueName: \"kubernetes.io/projected/9e4bee24-24f7-4e15-9ede-694573ea93d3-kube-api-access-cjcqx\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:05.677087 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.677049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w" event={"ID":"9e4bee24-24f7-4e15-9ede-694573ea93d3","Type":"ContainerDied","Data":"2258d68273570ef6198d09f12db11fad01d310237805356452e0116c890b625f"}
Sep 15 21:18:05.677087 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.677069    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwg2w"
Sep 15 21:18:05.677562 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.677120    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2258d68273570ef6198d09f12db11fad01d310237805356452e0116c890b625f"
Sep 15 21:18:05.679091 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.679061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c" event={"ID":"cf5de0c5-9beb-423a-8829-1a836e83c5c1","Type":"ContainerDied","Data":"92978f5d3bc2ef9606b99d1a94b748a257bce429ad26b1d44077b5f7a1c178f5"}
Sep 15 21:18:05.679223 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.679100    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd92c"
Sep 15 21:18:05.679223 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.679110    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92978f5d3bc2ef9606b99d1a94b748a257bce429ad26b1d44077b5f7a1c178f5"
Sep 15 21:18:05.811309 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:05.811283    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:06.000501 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.000414    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wff6\" (UniqueName: \"kubernetes.io/projected/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-kube-api-access-5wff6\") pod \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") "
Sep 15 21:18:06.000501 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.000498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-util\") pod \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") "
Sep 15 21:18:06.000692 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.000529    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-bundle\") pod \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\" (UID: \"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1\") "
Sep 15 21:18:06.001028 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.001002    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-bundle" (OuterVolumeSpecName: "bundle") pod "5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" (UID: "5fb0bd64-16a6-47e4-a8ea-5630517aa2a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:06.002620 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.002595    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-kube-api-access-5wff6" (OuterVolumeSpecName: "kube-api-access-5wff6") pod "5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" (UID: "5fb0bd64-16a6-47e4-a8ea-5630517aa2a1"). InnerVolumeSpecName "kube-api-access-5wff6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:06.038479 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.038435    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-util" (OuterVolumeSpecName: "util") pod "5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" (UID: "5fb0bd64-16a6-47e4-a8ea-5630517aa2a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:06.101273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.101233    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wff6\" (UniqueName: \"kubernetes.io/projected/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-kube-api-access-5wff6\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:06.101273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.101261    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-util\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:06.101273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.101271    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fb0bd64-16a6-47e4-a8ea-5630517aa2a1-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:06.686238 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.686206    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl"
Sep 15 21:18:06.686653 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.686204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck7dhl" event={"ID":"5fb0bd64-16a6-47e4-a8ea-5630517aa2a1","Type":"ContainerDied","Data":"4ec5651cc498c337cf4589bb00dc417e0ca2ddeed75fbc749077c03eb77569c7"}
Sep 15 21:18:06.686653 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:06.686333    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ec5651cc498c337cf4589bb00dc417e0ca2ddeed75fbc749077c03eb77569c7"
Sep 15 21:18:13.721373 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:13.721306    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:13.721836 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:13.721410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:13.726673 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:13.726642    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:14.735991 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:14.735961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-587c544fd5-x9xp7"
Sep 15 21:18:14.790192 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:14.790150    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64bcffdbf-zmg9f"]
Sep 15 21:18:39.618144 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618110    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"]
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618496    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5de0c5-9beb-423a-8829-1a836e83c5c1" containerName="util"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618511    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5de0c5-9beb-423a-8829-1a836e83c5c1" containerName="util"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618519    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" containerName="pull"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618524    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" containerName="pull"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618535    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25503b39-e3ef-4d11-b129-b9e8301e8b29" containerName="pull"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618540    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25503b39-e3ef-4d11-b129-b9e8301e8b29" containerName="pull"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618547    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5de0c5-9beb-423a-8829-1a836e83c5c1" containerName="extract"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618552    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5de0c5-9beb-423a-8829-1a836e83c5c1" containerName="extract"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618562    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5de0c5-9beb-423a-8829-1a836e83c5c1" containerName="pull"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618566    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5de0c5-9beb-423a-8829-1a836e83c5c1" containerName="pull"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618575    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25503b39-e3ef-4d11-b129-b9e8301e8b29" containerName="extract"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618580    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25503b39-e3ef-4d11-b129-b9e8301e8b29" containerName="extract"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618591    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25503b39-e3ef-4d11-b129-b9e8301e8b29" containerName="util"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618596    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25503b39-e3ef-4d11-b129-b9e8301e8b29" containerName="util"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618603    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e4bee24-24f7-4e15-9ede-694573ea93d3" containerName="pull"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618607    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e4bee24-24f7-4e15-9ede-694573ea93d3" containerName="pull"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618614    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e4bee24-24f7-4e15-9ede-694573ea93d3" containerName="extract"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618620    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e4bee24-24f7-4e15-9ede-694573ea93d3" containerName="extract"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618627    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" containerName="util"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618632    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" containerName="util"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618637    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e4bee24-24f7-4e15-9ede-694573ea93d3" containerName="util"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618642    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e4bee24-24f7-4e15-9ede-694573ea93d3" containerName="util"
Sep 15 21:18:39.618635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618647    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" containerName="extract"
Sep 15 21:18:39.619309 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618653    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" containerName="extract"
Sep 15 21:18:39.619309 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618710    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5de0c5-9beb-423a-8829-1a836e83c5c1" containerName="extract"
Sep 15 21:18:39.619309 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618719    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb0bd64-16a6-47e4-a8ea-5630517aa2a1" containerName="extract"
Sep 15 21:18:39.619309 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618724    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e4bee24-24f7-4e15-9ede-694573ea93d3" containerName="extract"
Sep 15 21:18:39.619309 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.618733    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25503b39-e3ef-4d11-b129-b9e8301e8b29" containerName="extract"
Sep 15 21:18:39.623643 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.623622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:39.627009 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.626981    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 21:18:39.627444 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.627428    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 21:18:39.628053 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.628036    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6z224\""
Sep 15 21:18:39.640133 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.640102    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"]
Sep 15 21:18:39.707604 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.707565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv8vx\" (UniqueName: \"kubernetes.io/projected/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-kube-api-access-cv8vx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-w7mps\" (UID: \"98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:39.707789 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.707655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-w7mps\" (UID: \"98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:39.808398 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.808339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-w7mps\" (UID: \"98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:39.808583 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.808431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv8vx\" (UniqueName: \"kubernetes.io/projected/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-kube-api-access-cv8vx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-w7mps\" (UID: \"98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:39.808774 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.808755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-w7mps\" (UID: \"98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:39.810298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.810264    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64bcffdbf-zmg9f" podUID="4a92efa3-20e1-458b-841f-2dff8843b882" containerName="console" containerID="cri-o://40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a" gracePeriod=15
Sep 15 21:18:39.824915 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.824881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv8vx\" (UniqueName: \"kubernetes.io/projected/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-kube-api-access-cv8vx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-w7mps\" (UID: \"98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:39.934485 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:39.934453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:40.062660 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.062630    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:18:40.212123 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.212036    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-serving-cert\") pod \"4a92efa3-20e1-458b-841f-2dff8843b882\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") "
Sep 15 21:18:40.212123 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.212090    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnvbg\" (UniqueName: \"kubernetes.io/projected/4a92efa3-20e1-458b-841f-2dff8843b882-kube-api-access-qnvbg\") pod \"4a92efa3-20e1-458b-841f-2dff8843b882\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") "
Sep 15 21:18:40.212335 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.212144    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-oauth-config\") pod \"4a92efa3-20e1-458b-841f-2dff8843b882\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") "
Sep 15 21:18:40.212335 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.212303    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-console-config\") pod \"4a92efa3-20e1-458b-841f-2dff8843b882\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") "
Sep 15 21:18:40.212485 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.212395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-oauth-serving-cert\") pod \"4a92efa3-20e1-458b-841f-2dff8843b882\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") "
Sep 15 21:18:40.212485 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.212429    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-service-ca\") pod \"4a92efa3-20e1-458b-841f-2dff8843b882\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") "
Sep 15 21:18:40.212485 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.212478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-trusted-ca-bundle\") pod \"4a92efa3-20e1-458b-841f-2dff8843b882\" (UID: \"4a92efa3-20e1-458b-841f-2dff8843b882\") "
Sep 15 21:18:40.212796 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.212741    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a92efa3-20e1-458b-841f-2dff8843b882" (UID: "4a92efa3-20e1-458b-841f-2dff8843b882"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:40.212796 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.212776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-console-config" (OuterVolumeSpecName: "console-config") pod "4a92efa3-20e1-458b-841f-2dff8843b882" (UID: "4a92efa3-20e1-458b-841f-2dff8843b882"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:40.213099 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.212890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a92efa3-20e1-458b-841f-2dff8843b882" (UID: "4a92efa3-20e1-458b-841f-2dff8843b882"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:40.213099 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.213012    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4a92efa3-20e1-458b-841f-2dff8843b882" (UID: "4a92efa3-20e1-458b-841f-2dff8843b882"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:40.214298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.214277    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a92efa3-20e1-458b-841f-2dff8843b882" (UID: "4a92efa3-20e1-458b-841f-2dff8843b882"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:40.214402 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.214279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a92efa3-20e1-458b-841f-2dff8843b882-kube-api-access-qnvbg" (OuterVolumeSpecName: "kube-api-access-qnvbg") pod "4a92efa3-20e1-458b-841f-2dff8843b882" (UID: "4a92efa3-20e1-458b-841f-2dff8843b882"). InnerVolumeSpecName "kube-api-access-qnvbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:40.214402 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.214299    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a92efa3-20e1-458b-841f-2dff8843b882" (UID: "4a92efa3-20e1-458b-841f-2dff8843b882"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:40.222729 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.222705    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"]
Sep 15 21:18:40.225582 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:18:40.225555    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a03bf2_e3e8_4e0e_b776_1b93a11cc0b7.slice/crio-cfac8a1b168fb1a10e42ee61773162bb528006752b47569630d4a9dcb65dc597 WatchSource:0}: Error finding container cfac8a1b168fb1a10e42ee61773162bb528006752b47569630d4a9dcb65dc597: Status 404 returned error can't find the container with id cfac8a1b168fb1a10e42ee61773162bb528006752b47569630d4a9dcb65dc597
Sep 15 21:18:40.313626 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.313587    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-trusted-ca-bundle\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:40.313626 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.313619    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-serving-cert\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:40.313626 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.313629    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnvbg\" (UniqueName: \"kubernetes.io/projected/4a92efa3-20e1-458b-841f-2dff8843b882-kube-api-access-qnvbg\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:40.313910 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.313638    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a92efa3-20e1-458b-841f-2dff8843b882-console-oauth-config\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:40.313910 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.313648    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-console-config\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:40.313910 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.313657    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-oauth-serving-cert\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:40.313910 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.313665    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a92efa3-20e1-458b-841f-2dff8843b882-service-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:40.628191 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.628159    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"]
Sep 15 21:18:40.635072 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.635037    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"]
Sep 15 21:18:40.661614 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.661576    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"]
Sep 15 21:18:40.662034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.662016    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a92efa3-20e1-458b-841f-2dff8843b882" containerName="console"
Sep 15 21:18:40.662117 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.662037    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a92efa3-20e1-458b-841f-2dff8843b882" containerName="console"
Sep 15 21:18:40.662169 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.662120    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a92efa3-20e1-458b-841f-2dff8843b882" containerName="console"
Sep 15 21:18:40.666659 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.666627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:18:40.685100 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.685064    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"]
Sep 15 21:18:40.818802 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.818722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cafcdebf-a3f2-4dd5-9010-03151346d4bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-xtrk8\" (UID: \"cafcdebf-a3f2-4dd5-9010-03151346d4bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:18:40.818802 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.818779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm4p6\" (UniqueName: \"kubernetes.io/projected/cafcdebf-a3f2-4dd5-9010-03151346d4bf-kube-api-access-sm4p6\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-xtrk8\" (UID: \"cafcdebf-a3f2-4dd5-9010-03151346d4bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:18:40.883707 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.883658    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a92efa3-20e1-458b-841f-2dff8843b882" containerID="40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a" exitCode=0
Sep 15 21:18:40.883904 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.883721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bcffdbf-zmg9f" event={"ID":"4a92efa3-20e1-458b-841f-2dff8843b882","Type":"ContainerDied","Data":"40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a"}
Sep 15 21:18:40.883904 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.883765    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bcffdbf-zmg9f"
Sep 15 21:18:40.883904 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.883781    2563 scope.go:117] "RemoveContainer" containerID="40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a"
Sep 15 21:18:40.883904 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.883768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bcffdbf-zmg9f" event={"ID":"4a92efa3-20e1-458b-841f-2dff8843b882","Type":"ContainerDied","Data":"38d1f6b91036e5bf43ebe63d226022de0cc4930701265e6533568c2bd656902a"}
Sep 15 21:18:40.897739 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.897675    2563 scope.go:117] "RemoveContainer" containerID="40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a"
Sep 15 21:18:40.898286 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:18:40.898260    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a\": container with ID starting with 40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a not found: ID does not exist" containerID="40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a"
Sep 15 21:18:40.898436 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.898299    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a"} err="failed to get container status \"40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a\": rpc error: code = NotFound desc = could not find container \"40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a\": container with ID starting with 40d83d84be246c56e150f5016bc5d5a8508df8048a409eb1dda760552969de0a not found: ID does not exist"
Sep 15 21:18:40.909460 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.909417    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64bcffdbf-zmg9f"]
Sep 15 21:18:40.915013 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.914965    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64bcffdbf-zmg9f"]
Sep 15 21:18:40.919803 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.919766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cafcdebf-a3f2-4dd5-9010-03151346d4bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-xtrk8\" (UID: \"cafcdebf-a3f2-4dd5-9010-03151346d4bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:18:40.919995 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.919813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm4p6\" (UniqueName: \"kubernetes.io/projected/cafcdebf-a3f2-4dd5-9010-03151346d4bf-kube-api-access-sm4p6\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-xtrk8\" (UID: \"cafcdebf-a3f2-4dd5-9010-03151346d4bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:18:40.920277 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.920252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cafcdebf-a3f2-4dd5-9010-03151346d4bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-xtrk8\" (UID: \"cafcdebf-a3f2-4dd5-9010-03151346d4bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:18:40.930735 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.930644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm4p6\" (UniqueName: \"kubernetes.io/projected/cafcdebf-a3f2-4dd5-9010-03151346d4bf-kube-api-access-sm4p6\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-xtrk8\" (UID: \"cafcdebf-a3f2-4dd5-9010-03151346d4bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:18:40.981313 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:40.981268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:18:41.281236 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:41.281206    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"]
Sep 15 21:18:41.283174 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:18:41.283140    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafcdebf_a3f2_4dd5_9010_03151346d4bf.slice/crio-1c43840aa9ebb9845da8c182ca12a1866e9ea4fd6aaad8c6aefd3b38948bfd1d WatchSource:0}: Error finding container 1c43840aa9ebb9845da8c182ca12a1866e9ea4fd6aaad8c6aefd3b38948bfd1d: Status 404 returned error can't find the container with id 1c43840aa9ebb9845da8c182ca12a1866e9ea4fd6aaad8c6aefd3b38948bfd1d
Sep 15 21:18:41.895050 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:41.895011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8" event={"ID":"cafcdebf-a3f2-4dd5-9010-03151346d4bf","Type":"ContainerStarted","Data":"1c43840aa9ebb9845da8c182ca12a1866e9ea4fd6aaad8c6aefd3b38948bfd1d"}
Sep 15 21:18:42.530734 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:42.530698    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a92efa3-20e1-458b-841f-2dff8843b882" path="/var/lib/kubelet/pods/4a92efa3-20e1-458b-841f-2dff8843b882/volumes"
Sep 15 21:18:44.918313 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:44.918268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8" event={"ID":"cafcdebf-a3f2-4dd5-9010-03151346d4bf","Type":"ContainerStarted","Data":"1751b9892371e5a2e54c2054dbd360745d1d0f534a605df973e6f291911b960a"}
Sep 15 21:18:44.918814 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:44.918466    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:18:44.920205 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:44.920177    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps" podUID="98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" containerName="manager" containerID="cri-o://9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1" gracePeriod=2
Sep 15 21:18:44.942804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:44.942751    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8" podStartSLOduration=1.940603069 podStartE2EDuration="4.942734626s" totalImagesPullingTime="3.002131557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:44.939473998 +0000 UTC m=+514.902531380" watchObservedRunningTime="2026-09-15 21:18:44.942734626 +0000 UTC m=+514.905792007"
Sep 15 21:18:45.164253 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.164228    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:45.166338 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:18:45.166309    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-w7mps\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:45.265361 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.265254    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv8vx\" (UniqueName: \"kubernetes.io/projected/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-kube-api-access-cv8vx\") pod \"98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7\" (UID: \"98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7\") "
Sep 15 21:18:45.265361 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.265302    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-extensions-socket-volume\") pod \"98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7\" (UID: \"98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7\") "
Sep 15 21:18:45.265600 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.265563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" (UID: "98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:45.267561 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.267534    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-kube-api-access-cv8vx" (OuterVolumeSpecName: "kube-api-access-cv8vx") pod "98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" (UID: "98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7"). InnerVolumeSpecName "kube-api-access-cv8vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:45.366556 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.366509    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv8vx\" (UniqueName: \"kubernetes.io/projected/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-kube-api-access-cv8vx\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:45.366556 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.366547    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7-extensions-socket-volume\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:18:45.927880 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.927844    2563 generic.go:356] "Generic (PLEG): container finished" podID="98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" containerID="9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1" exitCode=2
Sep 15 21:18:45.927880 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.927884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:45.928435 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.927949    2563 scope.go:117] "RemoveContainer" containerID="9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1"
Sep 15 21:18:45.930005 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:18:45.929976    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-w7mps\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:45.937927 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.937910    2563 scope.go:117] "RemoveContainer" containerID="9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1"
Sep 15 21:18:45.938265 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:18:45.938239    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-w7mps\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-w7mps"
Sep 15 21:18:45.938265 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:18:45.938255    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1\": container with ID starting with 9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1 not found: ID does not exist" containerID="9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1"
Sep 15 21:18:45.938376 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:45.938281    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1"} err="failed to get container status \"9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1\": rpc error: code = NotFound desc = could not find container \"9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1\": container with ID starting with 9d5fb22505dc73b431ba593fc27dd7bd0304966b1ee577544a3855f4c9c368b1 not found: ID does not exist"
Sep 15 21:18:46.526720 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:46.526686    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" path="/var/lib/kubelet/pods/98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7/volumes"
Sep 15 21:18:55.930752 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:18:55.930719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:19:08.866517 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:08.866440    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"]
Sep 15 21:19:08.866940 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:08.866647    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8" podUID="cafcdebf-a3f2-4dd5-9010-03151346d4bf" containerName="manager" containerID="cri-o://1751b9892371e5a2e54c2054dbd360745d1d0f534a605df973e6f291911b960a" gracePeriod=10
Sep 15 21:19:09.064394 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:09.064329    2563 generic.go:356] "Generic (PLEG): container finished" podID="cafcdebf-a3f2-4dd5-9010-03151346d4bf" containerID="1751b9892371e5a2e54c2054dbd360745d1d0f534a605df973e6f291911b960a" exitCode=0
Sep 15 21:19:09.064394 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:09.064381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8" event={"ID":"cafcdebf-a3f2-4dd5-9010-03151346d4bf","Type":"ContainerDied","Data":"1751b9892371e5a2e54c2054dbd360745d1d0f534a605df973e6f291911b960a"}
Sep 15 21:19:09.110465 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:09.110442    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:19:09.168415 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:09.168385    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cafcdebf-a3f2-4dd5-9010-03151346d4bf-extensions-socket-volume\") pod \"cafcdebf-a3f2-4dd5-9010-03151346d4bf\" (UID: \"cafcdebf-a3f2-4dd5-9010-03151346d4bf\") "
Sep 15 21:19:09.168593 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:09.168470    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm4p6\" (UniqueName: \"kubernetes.io/projected/cafcdebf-a3f2-4dd5-9010-03151346d4bf-kube-api-access-sm4p6\") pod \"cafcdebf-a3f2-4dd5-9010-03151346d4bf\" (UID: \"cafcdebf-a3f2-4dd5-9010-03151346d4bf\") "
Sep 15 21:19:09.168891 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:09.168866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafcdebf-a3f2-4dd5-9010-03151346d4bf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cafcdebf-a3f2-4dd5-9010-03151346d4bf" (UID: "cafcdebf-a3f2-4dd5-9010-03151346d4bf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:19:09.170609 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:09.170574    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafcdebf-a3f2-4dd5-9010-03151346d4bf-kube-api-access-sm4p6" (OuterVolumeSpecName: "kube-api-access-sm4p6") pod "cafcdebf-a3f2-4dd5-9010-03151346d4bf" (UID: "cafcdebf-a3f2-4dd5-9010-03151346d4bf"). InnerVolumeSpecName "kube-api-access-sm4p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:09.269423 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:09.269386    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm4p6\" (UniqueName: \"kubernetes.io/projected/cafcdebf-a3f2-4dd5-9010-03151346d4bf-kube-api-access-sm4p6\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:19:09.269423 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:09.269418    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cafcdebf-a3f2-4dd5-9010-03151346d4bf-extensions-socket-volume\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:19:10.072213 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:10.072185    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"
Sep 15 21:19:10.072753 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:10.072220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8" event={"ID":"cafcdebf-a3f2-4dd5-9010-03151346d4bf","Type":"ContainerDied","Data":"1c43840aa9ebb9845da8c182ca12a1866e9ea4fd6aaad8c6aefd3b38948bfd1d"}
Sep 15 21:19:10.072753 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:10.072291    2563 scope.go:117] "RemoveContainer" containerID="1751b9892371e5a2e54c2054dbd360745d1d0f534a605df973e6f291911b960a"
Sep 15 21:19:10.097444 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:10.097414    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"]
Sep 15 21:19:10.115158 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:10.115126    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-xtrk8"]
Sep 15 21:19:10.526683 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:10.526642    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cafcdebf-a3f2-4dd5-9010-03151346d4bf" path="/var/lib/kubelet/pods/cafcdebf-a3f2-4dd5-9010-03151346d4bf/volumes"
Sep 15 21:19:23.725815 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.725781    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6zmzf"]
Sep 15 21:19:23.726290 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.726167    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" containerName="manager"
Sep 15 21:19:23.726290 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.726180    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" containerName="manager"
Sep 15 21:19:23.726290 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.726207    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cafcdebf-a3f2-4dd5-9010-03151346d4bf" containerName="manager"
Sep 15 21:19:23.726290 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.726214    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cafcdebf-a3f2-4dd5-9010-03151346d4bf" containerName="manager"
Sep 15 21:19:23.726290 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.726279    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a03bf2-e3e8-4e0e-b776-1b93a11cc0b7" containerName="manager"
Sep 15 21:19:23.726290 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.726287    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cafcdebf-a3f2-4dd5-9010-03151346d4bf" containerName="manager"
Sep 15 21:19:23.731128 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.731110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:23.733629 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.733602    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 21:19:23.733629 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.733605    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 21:19:23.734409 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.734382    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 21:19:23.734544 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.734485    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gg6k6\""
Sep 15 21:19:23.737125 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.737102    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6zmzf"]
Sep 15 21:19:23.798724 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.798690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2700b4a-9d08-4f28-984d-d686f4a68a1e-config-file\") pod \"limitador-limitador-8588749977-6zmzf\" (UID: \"d2700b4a-9d08-4f28-984d-d686f4a68a1e\") " pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:23.798886 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.798742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdhf\" (UniqueName: \"kubernetes.io/projected/d2700b4a-9d08-4f28-984d-d686f4a68a1e-kube-api-access-bkdhf\") pod \"limitador-limitador-8588749977-6zmzf\" (UID: \"d2700b4a-9d08-4f28-984d-d686f4a68a1e\") " pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:23.822533 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.822498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6zmzf"]
Sep 15 21:19:23.899239 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.899205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdhf\" (UniqueName: \"kubernetes.io/projected/d2700b4a-9d08-4f28-984d-d686f4a68a1e-kube-api-access-bkdhf\") pod \"limitador-limitador-8588749977-6zmzf\" (UID: \"d2700b4a-9d08-4f28-984d-d686f4a68a1e\") " pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:23.899456 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.899288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2700b4a-9d08-4f28-984d-d686f4a68a1e-config-file\") pod \"limitador-limitador-8588749977-6zmzf\" (UID: \"d2700b4a-9d08-4f28-984d-d686f4a68a1e\") " pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:23.899933 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.899911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2700b4a-9d08-4f28-984d-d686f4a68a1e-config-file\") pod \"limitador-limitador-8588749977-6zmzf\" (UID: \"d2700b4a-9d08-4f28-984d-d686f4a68a1e\") " pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:23.908064 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:23.908040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdhf\" (UniqueName: \"kubernetes.io/projected/d2700b4a-9d08-4f28-984d-d686f4a68a1e-kube-api-access-bkdhf\") pod \"limitador-limitador-8588749977-6zmzf\" (UID: \"d2700b4a-9d08-4f28-984d-d686f4a68a1e\") " pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:24.043231 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:24.043124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:24.315783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:24.315756    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6zmzf"]
Sep 15 21:19:24.318411 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:19:24.318380    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2700b4a_9d08_4f28_984d_d686f4a68a1e.slice/crio-eaf092f2c4de49e3a5fc2e73bf8fd9be42165b8fca36b9d00225f5903e798817 WatchSource:0}: Error finding container eaf092f2c4de49e3a5fc2e73bf8fd9be42165b8fca36b9d00225f5903e798817: Status 404 returned error can't find the container with id eaf092f2c4de49e3a5fc2e73bf8fd9be42165b8fca36b9d00225f5903e798817
Sep 15 21:19:24.770094 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:24.770061    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-zb9xj"]
Sep 15 21:19:24.773410 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:24.773388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zb9xj"
Sep 15 21:19:24.775670 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:24.775647    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-467nr\""
Sep 15 21:19:24.783167 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:24.783095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zb9xj"]
Sep 15 21:19:24.909044 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:24.909001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6ng\" (UniqueName: \"kubernetes.io/projected/9c6295a5-34db-41aa-bbd0-52be684c4d2a-kube-api-access-cj6ng\") pod \"authorino-59c974bd48-zb9xj\" (UID: \"9c6295a5-34db-41aa-bbd0-52be684c4d2a\") " pod="kuadrant-system/authorino-59c974bd48-zb9xj"
Sep 15 21:19:25.009800 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:25.009753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6ng\" (UniqueName: \"kubernetes.io/projected/9c6295a5-34db-41aa-bbd0-52be684c4d2a-kube-api-access-cj6ng\") pod \"authorino-59c974bd48-zb9xj\" (UID: \"9c6295a5-34db-41aa-bbd0-52be684c4d2a\") " pod="kuadrant-system/authorino-59c974bd48-zb9xj"
Sep 15 21:19:25.023551 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:25.023475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6ng\" (UniqueName: \"kubernetes.io/projected/9c6295a5-34db-41aa-bbd0-52be684c4d2a-kube-api-access-cj6ng\") pod \"authorino-59c974bd48-zb9xj\" (UID: \"9c6295a5-34db-41aa-bbd0-52be684c4d2a\") " pod="kuadrant-system/authorino-59c974bd48-zb9xj"
Sep 15 21:19:25.085331 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:25.085285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zb9xj"
Sep 15 21:19:25.160533 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:25.160436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf" event={"ID":"d2700b4a-9d08-4f28-984d-d686f4a68a1e","Type":"ContainerStarted","Data":"eaf092f2c4de49e3a5fc2e73bf8fd9be42165b8fca36b9d00225f5903e798817"}
Sep 15 21:19:25.374321 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:25.374294    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zb9xj"]
Sep 15 21:19:25.376386 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:19:25.376316    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c6295a5_34db_41aa_bbd0_52be684c4d2a.slice/crio-525c74e7fec38249d48de5a467dac1de1aa2c21486d9da3e63a9469ee83d8a1d WatchSource:0}: Error finding container 525c74e7fec38249d48de5a467dac1de1aa2c21486d9da3e63a9469ee83d8a1d: Status 404 returned error can't find the container with id 525c74e7fec38249d48de5a467dac1de1aa2c21486d9da3e63a9469ee83d8a1d
Sep 15 21:19:26.167318 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:26.167272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf" event={"ID":"d2700b4a-9d08-4f28-984d-d686f4a68a1e","Type":"ContainerStarted","Data":"87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f"}
Sep 15 21:19:26.167790 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:26.167360    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:26.168563 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:26.168541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zb9xj" event={"ID":"9c6295a5-34db-41aa-bbd0-52be684c4d2a","Type":"ContainerStarted","Data":"525c74e7fec38249d48de5a467dac1de1aa2c21486d9da3e63a9469ee83d8a1d"}
Sep 15 21:19:26.186989 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:26.186923    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf" podStartSLOduration=2.370422481 podStartE2EDuration="3.186903398s" totalImagesPullingTime="816.480917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:26.186072087 +0000 UTC m=+556.149129468" watchObservedRunningTime="2026-09-15 21:19:26.186903398 +0000 UTC m=+556.149960781"
Sep 15 21:19:28.181425 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:28.181386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zb9xj" event={"ID":"9c6295a5-34db-41aa-bbd0-52be684c4d2a","Type":"ContainerStarted","Data":"dfc6ddb1f1c6df156feed5ebe700ad4865910e2a5f509965e973e3af2f46cd46"}
Sep 15 21:19:28.203881 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:28.203828    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-zb9xj" podStartSLOduration=2.312667858 podStartE2EDuration="4.203810787s" totalImagesPullingTime="1.891142929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:28.201143482 +0000 UTC m=+558.164200867" watchObservedRunningTime="2026-09-15 21:19:28.203810787 +0000 UTC m=+558.166868169"
Sep 15 21:19:37.176281 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:37.176242    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:37.999871 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:37.999828    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6zmzf"]
Sep 15 21:19:38.000132 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:38.000079    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf" podUID="d2700b4a-9d08-4f28-984d-d686f4a68a1e" containerName="limitador" containerID="cri-o://87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f" gracePeriod=30
Sep 15 21:19:38.542962 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:38.542935    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:38.736340 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:38.736302    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2700b4a-9d08-4f28-984d-d686f4a68a1e-config-file\") pod \"d2700b4a-9d08-4f28-984d-d686f4a68a1e\" (UID: \"d2700b4a-9d08-4f28-984d-d686f4a68a1e\") "
Sep 15 21:19:38.736554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:38.736464    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkdhf\" (UniqueName: \"kubernetes.io/projected/d2700b4a-9d08-4f28-984d-d686f4a68a1e-kube-api-access-bkdhf\") pod \"d2700b4a-9d08-4f28-984d-d686f4a68a1e\" (UID: \"d2700b4a-9d08-4f28-984d-d686f4a68a1e\") "
Sep 15 21:19:38.736756 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:38.736733    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2700b4a-9d08-4f28-984d-d686f4a68a1e-config-file" (OuterVolumeSpecName: "config-file") pod "d2700b4a-9d08-4f28-984d-d686f4a68a1e" (UID: "d2700b4a-9d08-4f28-984d-d686f4a68a1e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:19:38.738532 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:38.738507    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2700b4a-9d08-4f28-984d-d686f4a68a1e-kube-api-access-bkdhf" (OuterVolumeSpecName: "kube-api-access-bkdhf") pod "d2700b4a-9d08-4f28-984d-d686f4a68a1e" (UID: "d2700b4a-9d08-4f28-984d-d686f4a68a1e"). InnerVolumeSpecName "kube-api-access-bkdhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:38.837288 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:38.837248    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkdhf\" (UniqueName: \"kubernetes.io/projected/d2700b4a-9d08-4f28-984d-d686f4a68a1e-kube-api-access-bkdhf\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:19:38.837288 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:38.837281    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2700b4a-9d08-4f28-984d-d686f4a68a1e-config-file\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:19:39.243020 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:39.242986    2563 generic.go:356] "Generic (PLEG): container finished" podID="d2700b4a-9d08-4f28-984d-d686f4a68a1e" containerID="87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f" exitCode=0
Sep 15 21:19:39.243167 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:39.243026    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf"
Sep 15 21:19:39.243167 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:39.243064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf" event={"ID":"d2700b4a-9d08-4f28-984d-d686f4a68a1e","Type":"ContainerDied","Data":"87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f"}
Sep 15 21:19:39.243167 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:39.243117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6zmzf" event={"ID":"d2700b4a-9d08-4f28-984d-d686f4a68a1e","Type":"ContainerDied","Data":"eaf092f2c4de49e3a5fc2e73bf8fd9be42165b8fca36b9d00225f5903e798817"}
Sep 15 21:19:39.243167 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:39.243145    2563 scope.go:117] "RemoveContainer" containerID="87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f"
Sep 15 21:19:39.253139 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:39.253119    2563 scope.go:117] "RemoveContainer" containerID="87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f"
Sep 15 21:19:39.253464 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:19:39.253447    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f\": container with ID starting with 87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f not found: ID does not exist" containerID="87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f"
Sep 15 21:19:39.253510 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:39.253474    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f"} err="failed to get container status \"87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f\": rpc error: code = NotFound desc = could not find container \"87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f\": container with ID starting with 87d900770083d0946507e6e254c26cdb89569f38b4fd711a059a1108e64d932f not found: ID does not exist"
Sep 15 21:19:39.266556 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:39.266529    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6zmzf"]
Sep 15 21:19:39.271202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:39.271176    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6zmzf"]
Sep 15 21:19:40.529752 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:40.529707    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2700b4a-9d08-4f28-984d-d686f4a68a1e" path="/var/lib/kubelet/pods/d2700b4a-9d08-4f28-984d-d686f4a68a1e/volumes"
Sep 15 21:19:41.659971 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.659940    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"]
Sep 15 21:19:41.660438 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.660419    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2700b4a-9d08-4f28-984d-d686f4a68a1e" containerName="limitador"
Sep 15 21:19:41.660548 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.660446    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2700b4a-9d08-4f28-984d-d686f4a68a1e" containerName="limitador"
Sep 15 21:19:41.660548 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.660527    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2700b4a-9d08-4f28-984d-d686f4a68a1e" containerName="limitador"
Sep 15 21:19:41.664893 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.664874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"
Sep 15 21:19:41.668297 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.668264    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 21:19:41.668632 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.668614    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 15 21:19:41.668782 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.668761    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:19:41.670237 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.670218    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jvg55\""
Sep 15 21:19:41.692434 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.692395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"]
Sep 15 21:19:41.762769 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.762731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ztf\" (UniqueName: \"kubernetes.io/projected/bb8178b4-a139-4ffd-a84b-0434cb25075a-kube-api-access-l7ztf\") pod \"postgres-69fb4594cf-v7kd6\" (UID: \"bb8178b4-a139-4ffd-a84b-0434cb25075a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"
Sep 15 21:19:41.762957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.762792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb8178b4-a139-4ffd-a84b-0434cb25075a-data\") pod \"postgres-69fb4594cf-v7kd6\" (UID: \"bb8178b4-a139-4ffd-a84b-0434cb25075a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"
Sep 15 21:19:41.864054 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.864019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ztf\" (UniqueName: \"kubernetes.io/projected/bb8178b4-a139-4ffd-a84b-0434cb25075a-kube-api-access-l7ztf\") pod \"postgres-69fb4594cf-v7kd6\" (UID: \"bb8178b4-a139-4ffd-a84b-0434cb25075a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"
Sep 15 21:19:41.864250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.864084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb8178b4-a139-4ffd-a84b-0434cb25075a-data\") pod \"postgres-69fb4594cf-v7kd6\" (UID: \"bb8178b4-a139-4ffd-a84b-0434cb25075a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"
Sep 15 21:19:41.864493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.864471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb8178b4-a139-4ffd-a84b-0434cb25075a-data\") pod \"postgres-69fb4594cf-v7kd6\" (UID: \"bb8178b4-a139-4ffd-a84b-0434cb25075a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"
Sep 15 21:19:41.873692 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.873660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ztf\" (UniqueName: \"kubernetes.io/projected/bb8178b4-a139-4ffd-a84b-0434cb25075a-kube-api-access-l7ztf\") pod \"postgres-69fb4594cf-v7kd6\" (UID: \"bb8178b4-a139-4ffd-a84b-0434cb25075a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"
Sep 15 21:19:41.976699 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:41.976598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"
Sep 15 21:19:42.261118 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:42.261088    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"]
Sep 15 21:19:42.262106 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:19:42.262081    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8178b4_a139_4ffd_a84b_0434cb25075a.slice/crio-5344e0cab96fc838a510104dda321e65ebf31075bc1aa4e4e0b614b55118f3c6 WatchSource:0}: Error finding container 5344e0cab96fc838a510104dda321e65ebf31075bc1aa4e4e0b614b55118f3c6: Status 404 returned error can't find the container with id 5344e0cab96fc838a510104dda321e65ebf31075bc1aa4e4e0b614b55118f3c6
Sep 15 21:19:43.275454 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:43.275330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6" event={"ID":"bb8178b4-a139-4ffd-a84b-0434cb25075a","Type":"ContainerStarted","Data":"5344e0cab96fc838a510104dda321e65ebf31075bc1aa4e4e0b614b55118f3c6"}
Sep 15 21:19:47.303139 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:47.303029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6" event={"ID":"bb8178b4-a139-4ffd-a84b-0434cb25075a","Type":"ContainerStarted","Data":"49cfc0eee3e045059c667bc765f993a22def92aaab9e6b3515fc98fed4998146"}
Sep 15 21:19:47.303139 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:47.303108    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"
Sep 15 21:19:47.326111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:47.326063    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6" podStartSLOduration=1.553046184 podStartE2EDuration="6.326046544s" totalImagesPullingTime="4.77300036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:47.323059004 +0000 UTC m=+577.286116384" watchObservedRunningTime="2026-09-15 21:19:47.326046544 +0000 UTC m=+577.289103963"
Sep 15 21:19:53.337535 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:53.337493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v7kd6"
Sep 15 21:19:54.066022 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.065986    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4htvh"]
Sep 15 21:19:54.071923 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.071902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4htvh"
Sep 15 21:19:54.074215 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.074195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 21:19:54.078883 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.078856    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4htvh"]
Sep 15 21:19:54.152614 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.152580    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4htvh"]
Sep 15 21:19:54.152907 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:19:54.152885    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-h7zmx tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-4htvh" podUID="e0c06b09-99aa-406d-903e-152e9d1278cf"
Sep 15 21:19:54.183487 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.183450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e0c06b09-99aa-406d-903e-152e9d1278cf-tls-cert\") pod \"authorino-6d4dfc649f-4htvh\" (UID: \"e0c06b09-99aa-406d-903e-152e9d1278cf\") " pod="kuadrant-system/authorino-6d4dfc649f-4htvh"
Sep 15 21:19:54.183653 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.183579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zmx\" (UniqueName: \"kubernetes.io/projected/e0c06b09-99aa-406d-903e-152e9d1278cf-kube-api-access-h7zmx\") pod \"authorino-6d4dfc649f-4htvh\" (UID: \"e0c06b09-99aa-406d-903e-152e9d1278cf\") " pod="kuadrant-system/authorino-6d4dfc649f-4htvh"
Sep 15 21:19:54.284668 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.284630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zmx\" (UniqueName: \"kubernetes.io/projected/e0c06b09-99aa-406d-903e-152e9d1278cf-kube-api-access-h7zmx\") pod \"authorino-6d4dfc649f-4htvh\" (UID: \"e0c06b09-99aa-406d-903e-152e9d1278cf\") " pod="kuadrant-system/authorino-6d4dfc649f-4htvh"
Sep 15 21:19:54.284838 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.284687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e0c06b09-99aa-406d-903e-152e9d1278cf-tls-cert\") pod \"authorino-6d4dfc649f-4htvh\" (UID: \"e0c06b09-99aa-406d-903e-152e9d1278cf\") " pod="kuadrant-system/authorino-6d4dfc649f-4htvh"
Sep 15 21:19:54.287190 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.287170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e0c06b09-99aa-406d-903e-152e9d1278cf-tls-cert\") pod \"authorino-6d4dfc649f-4htvh\" (UID: \"e0c06b09-99aa-406d-903e-152e9d1278cf\") " pod="kuadrant-system/authorino-6d4dfc649f-4htvh"
Sep 15 21:19:54.292811 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.292785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zmx\" (UniqueName: \"kubernetes.io/projected/e0c06b09-99aa-406d-903e-152e9d1278cf-kube-api-access-h7zmx\") pod \"authorino-6d4dfc649f-4htvh\" (UID: \"e0c06b09-99aa-406d-903e-152e9d1278cf\") " pod="kuadrant-system/authorino-6d4dfc649f-4htvh"
Sep 15 21:19:54.340045 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.339966    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4htvh"
Sep 15 21:19:54.347944 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.347920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4htvh"
Sep 15 21:19:54.386037 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.386000    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7zmx\" (UniqueName: \"kubernetes.io/projected/e0c06b09-99aa-406d-903e-152e9d1278cf-kube-api-access-h7zmx\") pod \"e0c06b09-99aa-406d-903e-152e9d1278cf\" (UID: \"e0c06b09-99aa-406d-903e-152e9d1278cf\") "
Sep 15 21:19:54.386242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.386103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e0c06b09-99aa-406d-903e-152e9d1278cf-tls-cert\") pod \"e0c06b09-99aa-406d-903e-152e9d1278cf\" (UID: \"e0c06b09-99aa-406d-903e-152e9d1278cf\") "
Sep 15 21:19:54.388238 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.388209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c06b09-99aa-406d-903e-152e9d1278cf-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e0c06b09-99aa-406d-903e-152e9d1278cf" (UID: "e0c06b09-99aa-406d-903e-152e9d1278cf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:19:54.388238 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.388229    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c06b09-99aa-406d-903e-152e9d1278cf-kube-api-access-h7zmx" (OuterVolumeSpecName: "kube-api-access-h7zmx") pod "e0c06b09-99aa-406d-903e-152e9d1278cf" (UID: "e0c06b09-99aa-406d-903e-152e9d1278cf"). InnerVolumeSpecName "kube-api-access-h7zmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:54.487077 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.487041    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e0c06b09-99aa-406d-903e-152e9d1278cf-tls-cert\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:19:54.487077 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:54.487069    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7zmx\" (UniqueName: \"kubernetes.io/projected/e0c06b09-99aa-406d-903e-152e9d1278cf-kube-api-access-h7zmx\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:19:55.345624 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:55.345596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4htvh"
Sep 15 21:19:55.373851 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:55.373817    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4htvh"]
Sep 15 21:19:55.377507 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:55.377484    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4htvh"]
Sep 15 21:19:56.146445 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:56.146409    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zb9xj"]
Sep 15 21:19:56.146671 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:56.146644    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-zb9xj" podUID="9c6295a5-34db-41aa-bbd0-52be684c4d2a" containerName="authorino" containerID="cri-o://dfc6ddb1f1c6df156feed5ebe700ad4865910e2a5f509965e973e3af2f46cd46" gracePeriod=30
Sep 15 21:19:56.352716 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:56.352687    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c6295a5-34db-41aa-bbd0-52be684c4d2a" containerID="dfc6ddb1f1c6df156feed5ebe700ad4865910e2a5f509965e973e3af2f46cd46" exitCode=0
Sep 15 21:19:56.353089 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:56.352759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zb9xj" event={"ID":"9c6295a5-34db-41aa-bbd0-52be684c4d2a","Type":"ContainerDied","Data":"dfc6ddb1f1c6df156feed5ebe700ad4865910e2a5f509965e973e3af2f46cd46"}
Sep 15 21:19:56.526089 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:56.526012    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c06b09-99aa-406d-903e-152e9d1278cf" path="/var/lib/kubelet/pods/e0c06b09-99aa-406d-903e-152e9d1278cf/volumes"
Sep 15 21:19:56.958552 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:56.958525    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zb9xj"
Sep 15 21:19:57.013448 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:57.013416    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj6ng\" (UniqueName: \"kubernetes.io/projected/9c6295a5-34db-41aa-bbd0-52be684c4d2a-kube-api-access-cj6ng\") pod \"9c6295a5-34db-41aa-bbd0-52be684c4d2a\" (UID: \"9c6295a5-34db-41aa-bbd0-52be684c4d2a\") "
Sep 15 21:19:57.015490 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:57.015464    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6295a5-34db-41aa-bbd0-52be684c4d2a-kube-api-access-cj6ng" (OuterVolumeSpecName: "kube-api-access-cj6ng") pod "9c6295a5-34db-41aa-bbd0-52be684c4d2a" (UID: "9c6295a5-34db-41aa-bbd0-52be684c4d2a"). InnerVolumeSpecName "kube-api-access-cj6ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:57.115039 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:57.114936    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj6ng\" (UniqueName: \"kubernetes.io/projected/9c6295a5-34db-41aa-bbd0-52be684c4d2a-kube-api-access-cj6ng\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:19:57.362664 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:57.362626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zb9xj" event={"ID":"9c6295a5-34db-41aa-bbd0-52be684c4d2a","Type":"ContainerDied","Data":"525c74e7fec38249d48de5a467dac1de1aa2c21486d9da3e63a9469ee83d8a1d"}
Sep 15 21:19:57.362664 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:57.362660    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zb9xj"
Sep 15 21:19:57.363127 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:57.362690    2563 scope.go:117] "RemoveContainer" containerID="dfc6ddb1f1c6df156feed5ebe700ad4865910e2a5f509965e973e3af2f46cd46"
Sep 15 21:19:57.386634 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:57.386589    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zb9xj"]
Sep 15 21:19:57.394185 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:57.394045    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zb9xj"]
Sep 15 21:19:58.525782 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:19:58.525748    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c6295a5-34db-41aa-bbd0-52be684c4d2a" path="/var/lib/kubelet/pods/9c6295a5-34db-41aa-bbd0-52be684c4d2a/volumes"
Sep 15 21:20:00.845243 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.845214    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c5f897b4c-2lzsr"]
Sep 15 21:20:00.845634 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.845596    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c6295a5-34db-41aa-bbd0-52be684c4d2a" containerName="authorino"
Sep 15 21:20:00.845634 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.845609    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c6295a5-34db-41aa-bbd0-52be684c4d2a" containerName="authorino"
Sep 15 21:20:00.845709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.845674    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c6295a5-34db-41aa-bbd0-52be684c4d2a" containerName="authorino"
Sep 15 21:20:00.851444 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.851423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:00.854094 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.854072    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:20:00.854221 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.854205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-96fm8\""
Sep 15 21:20:00.854594 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.854579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:20:00.854849 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.854837    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:20:00.870519 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.870496    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c5f897b4c-2lzsr"]
Sep 15 21:20:00.953196 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.953159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-metrics-certs\") pod \"maas-controller-c5f897b4c-2lzsr\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") " pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:00.953407 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.953211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhgf\" (UniqueName: \"kubernetes.io/projected/22f0b56a-8a2a-42fd-a8fb-d65700eda000-kube-api-access-gbhgf\") pod \"maas-controller-c5f897b4c-2lzsr\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") " pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:00.953407 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:00.953298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-webhook-cert\") pod \"maas-controller-c5f897b4c-2lzsr\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") " pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:01.015272 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.015236    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-8jc8g"]
Sep 15 21:20:01.017680 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.017660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.029738 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.029714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-8jc8g"]
Sep 15 21:20:01.054298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.054269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-metrics-certs\") pod \"maas-controller-c5f897b4c-2lzsr\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") " pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:01.054501 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.054315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhgf\" (UniqueName: \"kubernetes.io/projected/22f0b56a-8a2a-42fd-a8fb-d65700eda000-kube-api-access-gbhgf\") pod \"maas-controller-c5f897b4c-2lzsr\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") " pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:01.054501 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.054368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-webhook-cert\") pod \"maas-controller-c5f897b4c-2lzsr\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") " pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:01.056753 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.056725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-webhook-cert\") pod \"maas-controller-c5f897b4c-2lzsr\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") " pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:01.056871 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.056761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-metrics-certs\") pod \"maas-controller-c5f897b4c-2lzsr\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") " pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:01.063651 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.063628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhgf\" (UniqueName: \"kubernetes.io/projected/22f0b56a-8a2a-42fd-a8fb-d65700eda000-kube-api-access-gbhgf\") pod \"maas-controller-c5f897b4c-2lzsr\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") " pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:01.155457 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.155420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-metrics-certs\") pod \"maas-controller-74f4d48d8b-8jc8g\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") " pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.155637 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.155549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-webhook-cert\") pod \"maas-controller-74f4d48d8b-8jc8g\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") " pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.155637 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.155593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5l4\" (UniqueName: \"kubernetes.io/projected/5c53dbcc-7d70-413b-ac17-797abca1f8df-kube-api-access-tq5l4\") pod \"maas-controller-74f4d48d8b-8jc8g\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") " pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.162860 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.162839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:01.256726 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.256690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-metrics-certs\") pod \"maas-controller-74f4d48d8b-8jc8g\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") " pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.256901 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.256852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-webhook-cert\") pod \"maas-controller-74f4d48d8b-8jc8g\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") " pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.256946 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.256914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5l4\" (UniqueName: \"kubernetes.io/projected/5c53dbcc-7d70-413b-ac17-797abca1f8df-kube-api-access-tq5l4\") pod \"maas-controller-74f4d48d8b-8jc8g\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") " pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.259199 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.259176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-webhook-cert\") pod \"maas-controller-74f4d48d8b-8jc8g\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") " pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.259445 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.259424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-metrics-certs\") pod \"maas-controller-74f4d48d8b-8jc8g\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") " pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.265332 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.265306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5l4\" (UniqueName: \"kubernetes.io/projected/5c53dbcc-7d70-413b-ac17-797abca1f8df-kube-api-access-tq5l4\") pod \"maas-controller-74f4d48d8b-8jc8g\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") " pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.327735 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.327697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:01.433808 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.433774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c5f897b4c-2lzsr"]
Sep 15 21:20:01.435187 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:20:01.435163    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f0b56a_8a2a_42fd_a8fb_d65700eda000.slice/crio-765a33e926277b26709fcdb48cea7f466c2f612c6b9f0e7e5efcf4854757c429 WatchSource:0}: Error finding container 765a33e926277b26709fcdb48cea7f466c2f612c6b9f0e7e5efcf4854757c429: Status 404 returned error can't find the container with id 765a33e926277b26709fcdb48cea7f466c2f612c6b9f0e7e5efcf4854757c429
Sep 15 21:20:01.603565 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:01.603536    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-8jc8g"]
Sep 15 21:20:01.605306 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:20:01.605279    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c53dbcc_7d70_413b_ac17_797abca1f8df.slice/crio-20133920e0be4bbde574ec3a78038f46f913ef565a205fb5fe285ddbbed8181d WatchSource:0}: Error finding container 20133920e0be4bbde574ec3a78038f46f913ef565a205fb5fe285ddbbed8181d: Status 404 returned error can't find the container with id 20133920e0be4bbde574ec3a78038f46f913ef565a205fb5fe285ddbbed8181d
Sep 15 21:20:02.395927 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:02.395890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5f897b4c-2lzsr" event={"ID":"22f0b56a-8a2a-42fd-a8fb-d65700eda000","Type":"ContainerStarted","Data":"765a33e926277b26709fcdb48cea7f466c2f612c6b9f0e7e5efcf4854757c429"}
Sep 15 21:20:02.397146 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:02.397120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g" event={"ID":"5c53dbcc-7d70-413b-ac17-797abca1f8df","Type":"ContainerStarted","Data":"20133920e0be4bbde574ec3a78038f46f913ef565a205fb5fe285ddbbed8181d"}
Sep 15 21:20:05.421568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:05.421524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g" event={"ID":"5c53dbcc-7d70-413b-ac17-797abca1f8df","Type":"ContainerStarted","Data":"2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21"}
Sep 15 21:20:05.422005 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:05.421746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:05.423152 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:05.423129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5f897b4c-2lzsr" event={"ID":"22f0b56a-8a2a-42fd-a8fb-d65700eda000","Type":"ContainerStarted","Data":"ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98"}
Sep 15 21:20:05.423287 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:05.423276    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:05.443671 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:05.443614    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g" podStartSLOduration=2.548834761 podStartE2EDuration="5.443592899s" totalImagesPullingTime="2.894758138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:05.440788113 +0000 UTC m=+595.403845494" watchObservedRunningTime="2026-09-15 21:20:05.443592899 +0000 UTC m=+595.406650280"
Sep 15 21:20:05.460588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:05.460512    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c5f897b4c-2lzsr" podStartSLOduration=2.401622201 podStartE2EDuration="5.460496562s" totalImagesPullingTime="3.058874361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:05.458946024 +0000 UTC m=+595.422003407" watchObservedRunningTime="2026-09-15 21:20:05.460496562 +0000 UTC m=+595.423553944"
Sep 15 21:20:16.433851 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.433819    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:16.434248 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.433897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:20:16.512222 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.512183    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c5f897b4c-2lzsr"]
Sep 15 21:20:16.512570 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.512515    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c5f897b4c-2lzsr" podUID="22f0b56a-8a2a-42fd-a8fb-d65700eda000" containerName="manager" containerID="cri-o://ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98" gracePeriod=30
Sep 15 21:20:16.753859 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.753837    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:16.805361 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.805318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-metrics-certs\") pod \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") "
Sep 15 21:20:16.805528 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.805402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-webhook-cert\") pod \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") "
Sep 15 21:20:16.805528 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.805433    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbhgf\" (UniqueName: \"kubernetes.io/projected/22f0b56a-8a2a-42fd-a8fb-d65700eda000-kube-api-access-gbhgf\") pod \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\" (UID: \"22f0b56a-8a2a-42fd-a8fb-d65700eda000\") "
Sep 15 21:20:16.807521 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.807486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "22f0b56a-8a2a-42fd-a8fb-d65700eda000" (UID: "22f0b56a-8a2a-42fd-a8fb-d65700eda000"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:20:16.807521 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.807488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f0b56a-8a2a-42fd-a8fb-d65700eda000-kube-api-access-gbhgf" (OuterVolumeSpecName: "kube-api-access-gbhgf") pod "22f0b56a-8a2a-42fd-a8fb-d65700eda000" (UID: "22f0b56a-8a2a-42fd-a8fb-d65700eda000"). InnerVolumeSpecName "kube-api-access-gbhgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:20:16.807686 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.807531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "22f0b56a-8a2a-42fd-a8fb-d65700eda000" (UID: "22f0b56a-8a2a-42fd-a8fb-d65700eda000"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:20:16.906153 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.906115    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-metrics-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:20:16.906153 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.906147    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f0b56a-8a2a-42fd-a8fb-d65700eda000-webhook-cert\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:20:16.906153 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:16.906158    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbhgf\" (UniqueName: \"kubernetes.io/projected/22f0b56a-8a2a-42fd-a8fb-d65700eda000-kube-api-access-gbhgf\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:20:17.493778 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:17.493740    2563 generic.go:356] "Generic (PLEG): container finished" podID="22f0b56a-8a2a-42fd-a8fb-d65700eda000" containerID="ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98" exitCode=0
Sep 15 21:20:17.494171 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:17.493787    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5f897b4c-2lzsr"
Sep 15 21:20:17.494171 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:17.493826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5f897b4c-2lzsr" event={"ID":"22f0b56a-8a2a-42fd-a8fb-d65700eda000","Type":"ContainerDied","Data":"ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98"}
Sep 15 21:20:17.494171 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:17.493884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5f897b4c-2lzsr" event={"ID":"22f0b56a-8a2a-42fd-a8fb-d65700eda000","Type":"ContainerDied","Data":"765a33e926277b26709fcdb48cea7f466c2f612c6b9f0e7e5efcf4854757c429"}
Sep 15 21:20:17.494171 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:17.493911    2563 scope.go:117] "RemoveContainer" containerID="ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98"
Sep 15 21:20:17.503606 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:17.503589    2563 scope.go:117] "RemoveContainer" containerID="ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98"
Sep 15 21:20:17.503868 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:20:17.503853    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98\": container with ID starting with ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98 not found: ID does not exist" containerID="ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98"
Sep 15 21:20:17.503908 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:17.503877    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98"} err="failed to get container status \"ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98\": rpc error: code = NotFound desc = could not find container \"ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98\": container with ID starting with ba259169d952b5e860223e6c7ffa3766570f0e401ae8fe4cf0792059bc7e2a98 not found: ID does not exist"
Sep 15 21:20:17.516182 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:17.516153    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c5f897b4c-2lzsr"]
Sep 15 21:20:17.521017 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:17.520992    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c5f897b4c-2lzsr"]
Sep 15 21:20:18.525599 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:18.525565    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f0b56a-8a2a-42fd-a8fb-d65700eda000" path="/var/lib/kubelet/pods/22f0b56a-8a2a-42fd-a8fb-d65700eda000/volumes"
Sep 15 21:20:38.288858 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.288816    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"]
Sep 15 21:20:38.289311 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.289186    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f0b56a-8a2a-42fd-a8fb-d65700eda000" containerName="manager"
Sep 15 21:20:38.289311 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.289198    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f0b56a-8a2a-42fd-a8fb-d65700eda000" containerName="manager"
Sep 15 21:20:38.289311 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.289268    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f0b56a-8a2a-42fd-a8fb-d65700eda000" containerName="manager"
Sep 15 21:20:38.292166 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.292149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.294570 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.294548    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 21:20:38.294570 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.294562    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4hmqz\""
Sep 15 21:20:38.294570 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.294549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 21:20:38.302908 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.302878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"]
Sep 15 21:20:38.399858 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.399807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-metrics-certs\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.400055 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.399879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e641850-54c7-47c8-a646-69fc7e89cff7-openshift-service-ca\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.400055 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.399906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7z9\" (UniqueName: \"kubernetes.io/projected/6e641850-54c7-47c8-a646-69fc7e89cff7-kube-api-access-8k7z9\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.400055 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.400024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-maas-api-tls\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.501308 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.501268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-metrics-certs\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.501515 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.501323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e641850-54c7-47c8-a646-69fc7e89cff7-openshift-service-ca\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.501515 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.501376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7z9\" (UniqueName: \"kubernetes.io/projected/6e641850-54c7-47c8-a646-69fc7e89cff7-kube-api-access-8k7z9\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.501515 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.501420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-maas-api-tls\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.501957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.501935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e641850-54c7-47c8-a646-69fc7e89cff7-openshift-service-ca\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.503739 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.503711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-maas-api-tls\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.503847 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.503805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-metrics-certs\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.509881 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.509858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7z9\" (UniqueName: \"kubernetes.io/projected/6e641850-54c7-47c8-a646-69fc7e89cff7-kube-api-access-8k7z9\") pod \"maas-api-5c78f66949-2jnj2\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") " pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.602873 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.602775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:38.881689 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.881658    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"]
Sep 15 21:20:38.883540 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:20:38.883512    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e641850_54c7_47c8_a646_69fc7e89cff7.slice/crio-a7d725d6da550c2fc62a8a0bec4d272d2539cac07aa74a3ad1b2fa1b824e420c WatchSource:0}: Error finding container a7d725d6da550c2fc62a8a0bec4d272d2539cac07aa74a3ad1b2fa1b824e420c: Status 404 returned error can't find the container with id a7d725d6da550c2fc62a8a0bec4d272d2539cac07aa74a3ad1b2fa1b824e420c
Sep 15 21:20:38.885121 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.885099    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:20:38.897575 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.897547    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"]
Sep 15 21:20:38.902148 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.902130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"
Sep 15 21:20:38.904116 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.904095    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 21:20:38.904271 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.904244    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tx2kg\""
Sep 15 21:20:38.911882 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:38.911860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"]
Sep 15 21:20:39.006299 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:39.006266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aab6349-f631-40d4-8e1b-6ad8fbab3f7a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-q7g9m\" (UID: \"5aab6349-f631-40d4-8e1b-6ad8fbab3f7a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"
Sep 15 21:20:39.006574 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:39.006436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgln\" (UniqueName: \"kubernetes.io/projected/5aab6349-f631-40d4-8e1b-6ad8fbab3f7a-kube-api-access-8kgln\") pod \"payload-processing-5b54b7fbb7-q7g9m\" (UID: \"5aab6349-f631-40d4-8e1b-6ad8fbab3f7a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"
Sep 15 21:20:39.107334 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:39.107287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgln\" (UniqueName: \"kubernetes.io/projected/5aab6349-f631-40d4-8e1b-6ad8fbab3f7a-kube-api-access-8kgln\") pod \"payload-processing-5b54b7fbb7-q7g9m\" (UID: \"5aab6349-f631-40d4-8e1b-6ad8fbab3f7a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"
Sep 15 21:20:39.107562 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:39.107413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aab6349-f631-40d4-8e1b-6ad8fbab3f7a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-q7g9m\" (UID: \"5aab6349-f631-40d4-8e1b-6ad8fbab3f7a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"
Sep 15 21:20:39.108186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:39.108168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aab6349-f631-40d4-8e1b-6ad8fbab3f7a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-q7g9m\" (UID: \"5aab6349-f631-40d4-8e1b-6ad8fbab3f7a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"
Sep 15 21:20:39.117499 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:39.117470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgln\" (UniqueName: \"kubernetes.io/projected/5aab6349-f631-40d4-8e1b-6ad8fbab3f7a-kube-api-access-8kgln\") pod \"payload-processing-5b54b7fbb7-q7g9m\" (UID: \"5aab6349-f631-40d4-8e1b-6ad8fbab3f7a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"
Sep 15 21:20:39.215474 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:39.215389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"
Sep 15 21:20:39.491437 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:39.491406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"]
Sep 15 21:20:39.492403 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:20:39.492377    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aab6349_f631_40d4_8e1b_6ad8fbab3f7a.slice/crio-4168f63bbb71a2840b2cd55baf6bb15aeae7e281bd352445fc852ac07fc7cea4 WatchSource:0}: Error finding container 4168f63bbb71a2840b2cd55baf6bb15aeae7e281bd352445fc852ac07fc7cea4: Status 404 returned error can't find the container with id 4168f63bbb71a2840b2cd55baf6bb15aeae7e281bd352445fc852ac07fc7cea4
Sep 15 21:20:39.616404 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:39.616372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m" event={"ID":"5aab6349-f631-40d4-8e1b-6ad8fbab3f7a","Type":"ContainerStarted","Data":"4168f63bbb71a2840b2cd55baf6bb15aeae7e281bd352445fc852ac07fc7cea4"}
Sep 15 21:20:39.617660 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:39.617626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2" event={"ID":"6e641850-54c7-47c8-a646-69fc7e89cff7","Type":"ContainerStarted","Data":"a7d725d6da550c2fc62a8a0bec4d272d2539cac07aa74a3ad1b2fa1b824e420c"}
Sep 15 21:20:41.631221 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:41.631181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2" event={"ID":"6e641850-54c7-47c8-a646-69fc7e89cff7","Type":"ContainerStarted","Data":"dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934"}
Sep 15 21:20:41.631703 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:41.631285    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:41.632886 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:41.632866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m" event={"ID":"5aab6349-f631-40d4-8e1b-6ad8fbab3f7a","Type":"ContainerStarted","Data":"43ddc2207614ae23463a9c63279366543bf29a21777e1df673769e55d81f8bd4"}
Sep 15 21:20:41.632996 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:41.632985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"
Sep 15 21:20:41.649455 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:41.649409    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2" podStartSLOduration=1.563840654 podStartE2EDuration="3.649395506s" totalImagesPullingTime="2.085554852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:41.647719176 +0000 UTC m=+631.610776561" watchObservedRunningTime="2026-09-15 21:20:41.649395506 +0000 UTC m=+631.612452887"
Sep 15 21:20:41.664445 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:41.664407    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m" podStartSLOduration=2.185175452 podStartE2EDuration="3.664394403s" totalImagesPullingTime="1.479218951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:41.663225637 +0000 UTC m=+631.626283019" watchObservedRunningTime="2026-09-15 21:20:41.664394403 +0000 UTC m=+631.627451787"
Sep 15 21:20:47.644813 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:47.644782    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:20:52.641799 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:20:52.641767    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-q7g9m"
Sep 15 21:21:20.311653 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.311607    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"]
Sep 15 21:21:20.317025 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.316994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.320641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.320616    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 15 21:21:20.320792 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.320674    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mjjfq\""
Sep 15 21:21:20.320792 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.320616    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 21:21:20.320792 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.320629    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 21:21:20.326251 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.326225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"]
Sep 15 21:21:20.380949 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.380912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.381155 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.380991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.381155 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.381024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.381155 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.381096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.381155 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.381125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.381443 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.381158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxls\" (UniqueName: \"kubernetes.io/projected/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-kube-api-access-qnxls\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.482234 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.482195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.482456 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.482248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.482456 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.482269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.482456 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.482299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxls\" (UniqueName: \"kubernetes.io/projected/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-kube-api-access-qnxls\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.482456 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.482368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.482456 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.482420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.482725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.482698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.482819 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.482796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.482889 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.482870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.484721 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.484697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.484887 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.484871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.491196 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.491170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxls\" (UniqueName: \"kubernetes.io/projected/97a280e4-ee73-44ad-8d9a-e4284eaf43f3-kube-api-access-qnxls\") pod \"e2e-distinct-simulated-kserve-59975d69f9-svkkf\" (UID: \"97a280e4-ee73-44ad-8d9a-e4284eaf43f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.628202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.628165    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:20.907408 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:20.907378    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"]
Sep 15 21:21:20.908798 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:21:20.908776    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a280e4_ee73_44ad_8d9a_e4284eaf43f3.slice/crio-5783fe0711239d66acc62985394c2a081a44cecf14e8155a497f9852073e4440 WatchSource:0}: Error finding container 5783fe0711239d66acc62985394c2a081a44cecf14e8155a497f9852073e4440: Status 404 returned error can't find the container with id 5783fe0711239d66acc62985394c2a081a44cecf14e8155a497f9852073e4440
Sep 15 21:21:21.859176 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:21.859099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf" event={"ID":"97a280e4-ee73-44ad-8d9a-e4284eaf43f3","Type":"ContainerStarted","Data":"5783fe0711239d66acc62985394c2a081a44cecf14e8155a497f9852073e4440"}
Sep 15 21:21:22.866040 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:22.865994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf" event={"ID":"97a280e4-ee73-44ad-8d9a-e4284eaf43f3","Type":"ContainerStarted","Data":"679bab75d126b91002a269b1de5c6e1f591d35d58af497c33d166ce74c119c33"}
Sep 15 21:21:22.884684 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:22.884614    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf" podStartSLOduration=1.830690892 podStartE2EDuration="2.884597801s" totalImagesPullingTime="1.053906909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:22.883531265 +0000 UTC m=+672.846588645" watchObservedRunningTime="2026-09-15 21:21:22.884597801 +0000 UTC m=+672.847655182"
Sep 15 21:21:30.629064 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:30.629021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:30.629541 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:30.629086    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:30.641419 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:30.641392    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:21:30.921552 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:21:30.921470    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-svkkf"
Sep 15 21:22:20.114048 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.113961    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"]
Sep 15 21:22:20.117478 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.117458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.119910 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.119889    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 15 21:22:20.128964 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.128938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"]
Sep 15 21:22:20.220689 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.220664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22f8k\" (UniqueName: \"kubernetes.io/projected/fbc46b7c-9dce-444d-8a09-cd2d596a698e-kube-api-access-22f8k\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.220851 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.220705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.220851 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.220776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.220851 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.220811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.220954 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.220856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.220954 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.220887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc46b7c-9dce-444d-8a09-cd2d596a698e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.321565 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.321536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.321565 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.321566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc46b7c-9dce-444d-8a09-cd2d596a698e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.321756 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.321594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22f8k\" (UniqueName: \"kubernetes.io/projected/fbc46b7c-9dce-444d-8a09-cd2d596a698e-kube-api-access-22f8k\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.321756 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.321623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.321756 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.321657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.321756 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.321686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.322097 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.322075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.322162 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.322092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.322162 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.322128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.323768 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.323748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc46b7c-9dce-444d-8a09-cd2d596a698e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.324014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.323997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc46b7c-9dce-444d-8a09-cd2d596a698e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.330311 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.330291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22f8k\" (UniqueName: \"kubernetes.io/projected/fbc46b7c-9dce-444d-8a09-cd2d596a698e-kube-api-access-22f8k\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bbscv\" (UID: \"fbc46b7c-9dce-444d-8a09-cd2d596a698e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.429422 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.429404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:20.703186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:20.703159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"]
Sep 15 21:22:20.704929 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:22:20.704901    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc46b7c_9dce_444d_8a09_cd2d596a698e.slice/crio-8d41198e8b9e469d9d6f543920b19a7bdc8539f8a6dc1d1fb81954cb22db5145 WatchSource:0}: Error finding container 8d41198e8b9e469d9d6f543920b19a7bdc8539f8a6dc1d1fb81954cb22db5145: Status 404 returned error can't find the container with id 8d41198e8b9e469d9d6f543920b19a7bdc8539f8a6dc1d1fb81954cb22db5145
Sep 15 21:22:21.188696 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:21.188666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv" event={"ID":"fbc46b7c-9dce-444d-8a09-cd2d596a698e","Type":"ContainerStarted","Data":"a7087066143aea8a0f0994953c419b60f8e63aedede1714869381a4a42dd48c5"}
Sep 15 21:22:21.189038 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:21.188737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv" event={"ID":"fbc46b7c-9dce-444d-8a09-cd2d596a698e","Type":"ContainerStarted","Data":"8d41198e8b9e469d9d6f543920b19a7bdc8539f8a6dc1d1fb81954cb22db5145"}
Sep 15 21:22:21.223830 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:21.223750    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv" podStartSLOduration=1.020090346 podStartE2EDuration="1.223736462s" totalImagesPullingTime="203.646116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:21.222464494 +0000 UTC m=+731.185521886" watchObservedRunningTime="2026-09-15 21:22:21.223736462 +0000 UTC m=+731.186793844"
Sep 15 21:22:28.203305 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.203267    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"]
Sep 15 21:22:28.207416 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.207393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.209962 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.209936    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 21:22:28.222003 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.221975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"]
Sep 15 21:22:28.294948 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.294912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.294948 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.294955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dabdded8-ab58-45bc-b4e8-0c982279812e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.295166 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.295059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.295166 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.295083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.295166 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.295108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rqg\" (UniqueName: \"kubernetes.io/projected/dabdded8-ab58-45bc-b4e8-0c982279812e-kube-api-access-h2rqg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.295267 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.295178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.396610 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.396573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.396793 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.396641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.396793 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.396665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dabdded8-ab58-45bc-b4e8-0c982279812e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.396793 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.396702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.396793 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.396726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.397014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.396902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rqg\" (UniqueName: \"kubernetes.io/projected/dabdded8-ab58-45bc-b4e8-0c982279812e-kube-api-access-h2rqg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.397075 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.397005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.397129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.397098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.397129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.397118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.398931 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.398911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dabdded8-ab58-45bc-b4e8-0c982279812e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.399260 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.399245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dabdded8-ab58-45bc-b4e8-0c982279812e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.406427 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.406404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rqg\" (UniqueName: \"kubernetes.io/projected/dabdded8-ab58-45bc-b4e8-0c982279812e-kube-api-access-h2rqg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-64dqf\" (UID: \"dabdded8-ab58-45bc-b4e8-0c982279812e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.517618 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.517521    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:28.792584 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:28.792559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"]
Sep 15 21:22:28.794234 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:22:28.794207    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddabdded8_ab58_45bc_b4e8_0c982279812e.slice/crio-caf3304741b0bf12ab05422db145a548978a7b19a5a2651d56cdaf5130a832fa WatchSource:0}: Error finding container caf3304741b0bf12ab05422db145a548978a7b19a5a2651d56cdaf5130a832fa: Status 404 returned error can't find the container with id caf3304741b0bf12ab05422db145a548978a7b19a5a2651d56cdaf5130a832fa
Sep 15 21:22:29.236057 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:29.236025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf" event={"ID":"dabdded8-ab58-45bc-b4e8-0c982279812e","Type":"ContainerStarted","Data":"12c7e8d9336ce93eeb29bf160e16644fca52a640c2fd54842091278215e5302c"}
Sep 15 21:22:29.236511 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:29.236066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf" event={"ID":"dabdded8-ab58-45bc-b4e8-0c982279812e","Type":"ContainerStarted","Data":"caf3304741b0bf12ab05422db145a548978a7b19a5a2651d56cdaf5130a832fa"}
Sep 15 21:22:29.253989 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:29.253941    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf" podStartSLOduration=1.043451804 podStartE2EDuration="1.253924758s" totalImagesPullingTime="210.472954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:29.252295743 +0000 UTC m=+739.215353139" watchObservedRunningTime="2026-09-15 21:22:29.253924758 +0000 UTC m=+739.216982158"
Sep 15 21:22:30.429879 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:30.429848    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:30.430268 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:30.429895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:30.442108 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:30.442085    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:31.258058 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:31.258033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bbscv"
Sep 15 21:22:38.517699 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:38.517656    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:38.518093 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:38.517713    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:38.530503 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:38.530477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:22:39.301755 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:22:39.301726    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-64dqf"
Sep 15 21:23:32.780148 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.780090    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84587fcc9d-kgk94"]
Sep 15 21:23:32.783747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.783729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84587fcc9d-kgk94"
Sep 15 21:23:32.785925 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.785905    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 21:23:32.786828 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.786808    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 21:23:32.786904 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.786852    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-467nr\""
Sep 15 21:23:32.786904 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.786866    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 21:23:32.794806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.794781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84587fcc9d-kgk94"]
Sep 15 21:23:32.884590 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.884546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjfz\" (UniqueName: \"kubernetes.io/projected/48b7c0e0-7f46-4e03-bce8-60c82285ac30-kube-api-access-hgjfz\") pod \"authorino-84587fcc9d-kgk94\" (UID: \"48b7c0e0-7f46-4e03-bce8-60c82285ac30\") " pod="kuadrant-system/authorino-84587fcc9d-kgk94"
Sep 15 21:23:32.884769 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.884617    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/48b7c0e0-7f46-4e03-bce8-60c82285ac30-tls-cert\") pod \"authorino-84587fcc9d-kgk94\" (UID: \"48b7c0e0-7f46-4e03-bce8-60c82285ac30\") " pod="kuadrant-system/authorino-84587fcc9d-kgk94"
Sep 15 21:23:32.985898 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.985803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjfz\" (UniqueName: \"kubernetes.io/projected/48b7c0e0-7f46-4e03-bce8-60c82285ac30-kube-api-access-hgjfz\") pod \"authorino-84587fcc9d-kgk94\" (UID: \"48b7c0e0-7f46-4e03-bce8-60c82285ac30\") " pod="kuadrant-system/authorino-84587fcc9d-kgk94"
Sep 15 21:23:32.985898 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.985888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/48b7c0e0-7f46-4e03-bce8-60c82285ac30-tls-cert\") pod \"authorino-84587fcc9d-kgk94\" (UID: \"48b7c0e0-7f46-4e03-bce8-60c82285ac30\") " pod="kuadrant-system/authorino-84587fcc9d-kgk94"
Sep 15 21:23:32.988396 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.988364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/48b7c0e0-7f46-4e03-bce8-60c82285ac30-tls-cert\") pod \"authorino-84587fcc9d-kgk94\" (UID: \"48b7c0e0-7f46-4e03-bce8-60c82285ac30\") " pod="kuadrant-system/authorino-84587fcc9d-kgk94"
Sep 15 21:23:32.994550 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:32.994524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjfz\" (UniqueName: \"kubernetes.io/projected/48b7c0e0-7f46-4e03-bce8-60c82285ac30-kube-api-access-hgjfz\") pod \"authorino-84587fcc9d-kgk94\" (UID: \"48b7c0e0-7f46-4e03-bce8-60c82285ac30\") " pod="kuadrant-system/authorino-84587fcc9d-kgk94"
Sep 15 21:23:33.096284 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:33.096196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84587fcc9d-kgk94"
Sep 15 21:23:33.367163 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:33.367134    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84587fcc9d-kgk94"]
Sep 15 21:23:33.368277 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:23:33.368253    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48b7c0e0_7f46_4e03_bce8_60c82285ac30.slice/crio-251e315a8862a311f446238bb5f9ba7a82a560094bbbbaa0b0337a53cf9d8449 WatchSource:0}: Error finding container 251e315a8862a311f446238bb5f9ba7a82a560094bbbbaa0b0337a53cf9d8449: Status 404 returned error can't find the container with id 251e315a8862a311f446238bb5f9ba7a82a560094bbbbaa0b0337a53cf9d8449
Sep 15 21:23:33.593741 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:33.593708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84587fcc9d-kgk94" event={"ID":"48b7c0e0-7f46-4e03-bce8-60c82285ac30","Type":"ContainerStarted","Data":"251e315a8862a311f446238bb5f9ba7a82a560094bbbbaa0b0337a53cf9d8449"}
Sep 15 21:23:34.603176 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:34.603090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84587fcc9d-kgk94" event={"ID":"48b7c0e0-7f46-4e03-bce8-60c82285ac30","Type":"ContainerStarted","Data":"f876c596f7562eb768638f987651b571ed48b2495e20a8ba4ca1d85fa0f3ab3f"}
Sep 15 21:23:34.623028 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:34.622969    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84587fcc9d-kgk94" podStartSLOduration=1.690482608 podStartE2EDuration="2.622949406s" totalImagesPullingTime="932.466798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:34.620579945 +0000 UTC m=+804.583637328" watchObservedRunningTime="2026-09-15 21:23:34.622949406 +0000 UTC m=+804.586006788"
Sep 15 21:23:54.247582 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:54.247548    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7863-b9e177"]
Sep 15 21:23:54.251227 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:54.251207    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-b9e177"
Sep 15 21:23:54.253533 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:54.253517    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncvz5\""
Sep 15 21:23:54.257979 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:54.257956    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-b9e177"]
Sep 15 21:23:54.391923 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:54.391888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5vn8\" (UniqueName: \"kubernetes.io/projected/e0ee9b1a-fa70-4e3d-bcae-a56df780c483-kube-api-access-b5vn8\") pod \"test-curl-7863-b9e177\" (UID: \"e0ee9b1a-fa70-4e3d-bcae-a56df780c483\") " pod="opendatahub/test-curl-7863-b9e177"
Sep 15 21:23:54.492951 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:54.492920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5vn8\" (UniqueName: \"kubernetes.io/projected/e0ee9b1a-fa70-4e3d-bcae-a56df780c483-kube-api-access-b5vn8\") pod \"test-curl-7863-b9e177\" (UID: \"e0ee9b1a-fa70-4e3d-bcae-a56df780c483\") " pod="opendatahub/test-curl-7863-b9e177"
Sep 15 21:23:54.506000 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:54.505935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5vn8\" (UniqueName: \"kubernetes.io/projected/e0ee9b1a-fa70-4e3d-bcae-a56df780c483-kube-api-access-b5vn8\") pod \"test-curl-7863-b9e177\" (UID: \"e0ee9b1a-fa70-4e3d-bcae-a56df780c483\") " pod="opendatahub/test-curl-7863-b9e177"
Sep 15 21:23:54.561975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:54.561936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-b9e177"
Sep 15 21:23:54.833106 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:54.833080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-b9e177"]
Sep 15 21:23:54.834335 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:23:54.834307    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0ee9b1a_fa70_4e3d_bcae_a56df780c483.slice/crio-2a2dac825eed19969f56c358151bab5d099cceac938e40ba42ddbc2522376e15 WatchSource:0}: Error finding container 2a2dac825eed19969f56c358151bab5d099cceac938e40ba42ddbc2522376e15: Status 404 returned error can't find the container with id 2a2dac825eed19969f56c358151bab5d099cceac938e40ba42ddbc2522376e15
Sep 15 21:23:55.732365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:55.732312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-b9e177" event={"ID":"e0ee9b1a-fa70-4e3d-bcae-a56df780c483","Type":"ContainerStarted","Data":"2a2dac825eed19969f56c358151bab5d099cceac938e40ba42ddbc2522376e15"}
Sep 15 21:23:57.745273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:57.745239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-b9e177" event={"ID":"e0ee9b1a-fa70-4e3d-bcae-a56df780c483","Type":"ContainerStarted","Data":"4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267"}
Sep 15 21:23:57.759905 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:57.759855    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7863-b9e177" podStartSLOduration=1.5649527619999999 podStartE2EDuration="3.75984029s" totalImagesPullingTime="2.194887528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:57.758818193 +0000 UTC m=+827.721875573" watchObservedRunningTime="2026-09-15 21:23:57.75984029 +0000 UTC m=+827.722897718"
Sep 15 21:23:58.343911 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.343878    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7863-b9e177"]
Sep 15 21:23:58.350863 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.350835    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7863-b9e177"]
Sep 15 21:23:58.465213 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.465139    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7863-74252b"]
Sep 15 21:23:58.465581 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.465568    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ee9b1a-fa70-4e3d-bcae-a56df780c483" containerName="test-curl-7863-b9e177"
Sep 15 21:23:58.465635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.465583    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ee9b1a-fa70-4e3d-bcae-a56df780c483" containerName="test-curl-7863-b9e177"
Sep 15 21:23:58.465673 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.465652    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0ee9b1a-fa70-4e3d-bcae-a56df780c483" containerName="test-curl-7863-b9e177"
Sep 15 21:23:58.468928 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.468912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-74252b"
Sep 15 21:23:58.473454 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.473430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-74252b"]
Sep 15 21:23:58.633681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.633651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djbpv\" (UniqueName: \"kubernetes.io/projected/2a312e60-e45a-4cf1-b3d2-77c7e66791eb-kube-api-access-djbpv\") pod \"test-curl-7863-74252b\" (UID: \"2a312e60-e45a-4cf1-b3d2-77c7e66791eb\") " pod="opendatahub/test-curl-7863-74252b"
Sep 15 21:23:58.734886 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.734795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djbpv\" (UniqueName: \"kubernetes.io/projected/2a312e60-e45a-4cf1-b3d2-77c7e66791eb-kube-api-access-djbpv\") pod \"test-curl-7863-74252b\" (UID: \"2a312e60-e45a-4cf1-b3d2-77c7e66791eb\") " pod="opendatahub/test-curl-7863-74252b"
Sep 15 21:23:58.743177 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.743142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djbpv\" (UniqueName: \"kubernetes.io/projected/2a312e60-e45a-4cf1-b3d2-77c7e66791eb-kube-api-access-djbpv\") pod \"test-curl-7863-74252b\" (UID: \"2a312e60-e45a-4cf1-b3d2-77c7e66791eb\") " pod="opendatahub/test-curl-7863-74252b"
Sep 15 21:23:58.751552 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:23:58.751526    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7863-b9e177" container="test-curl-7863-b9e177"
Sep 15 21:23:58.779968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.779923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-74252b"
Sep 15 21:23:58.821021 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.820987    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"]
Sep 15 21:23:58.826812 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.826782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:58.829121 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.829100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-cfc1f4-serving-cert\""
Sep 15 21:23:58.836057 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.835913    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"]
Sep 15 21:23:58.936372 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.936309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6p6\" (UniqueName: \"kubernetes.io/projected/5a41def5-0a96-49cb-95fd-fd46b7743ef3-kube-api-access-6c6p6\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:58.936561 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.936442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-metrics-certs\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:58.936561 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.936499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-maas-api-tls\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:58.936561 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:58.936537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a41def5-0a96-49cb-95fd-fd46b7743ef3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:59.037386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.037274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6p6\" (UniqueName: \"kubernetes.io/projected/5a41def5-0a96-49cb-95fd-fd46b7743ef3-kube-api-access-6c6p6\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:59.037386 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.037355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-metrics-certs\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:59.037579 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.037420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-maas-api-tls\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:59.037579 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.037465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a41def5-0a96-49cb-95fd-fd46b7743ef3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:59.038196 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.038171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a41def5-0a96-49cb-95fd-fd46b7743ef3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:59.039837 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.039814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-maas-api-tls\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:59.039927 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.039844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-metrics-certs\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:59.045885 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.045863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6p6\" (UniqueName: \"kubernetes.io/projected/5a41def5-0a96-49cb-95fd-fd46b7743ef3-kube-api-access-6c6p6\") pod \"maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:59.061848 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.061826    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-74252b"]
Sep 15 21:23:59.063175 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:23:59.063147    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a312e60_e45a_4cf1_b3d2_77c7e66791eb.slice/crio-9371ae97587b417fb9da743e1ec3f923698fbfe2e141540642c67c66fb8cd0a8 WatchSource:0}: Error finding container 9371ae97587b417fb9da743e1ec3f923698fbfe2e141540642c67c66fb8cd0a8: Status 404 returned error can't find the container with id 9371ae97587b417fb9da743e1ec3f923698fbfe2e141540642c67c66fb8cd0a8
Sep 15 21:23:59.142300 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.142266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:23:59.419471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.419442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"]
Sep 15 21:23:59.420760 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:23:59.420732    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a41def5_0a96_49cb_95fd_fd46b7743ef3.slice/crio-5fb08607c147e09bea9b6a8394b03875a9717b0d66873f9b307629d432fd1941 WatchSource:0}: Error finding container 5fb08607c147e09bea9b6a8394b03875a9717b0d66873f9b307629d432fd1941: Status 404 returned error can't find the container with id 5fb08607c147e09bea9b6a8394b03875a9717b0d66873f9b307629d432fd1941
Sep 15 21:23:59.759570 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.759487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-74252b" event={"ID":"2a312e60-e45a-4cf1-b3d2-77c7e66791eb","Type":"ContainerStarted","Data":"64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3"}
Sep 15 21:23:59.759570 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.759532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-74252b" event={"ID":"2a312e60-e45a-4cf1-b3d2-77c7e66791eb","Type":"ContainerStarted","Data":"9371ae97587b417fb9da743e1ec3f923698fbfe2e141540642c67c66fb8cd0a8"}
Sep 15 21:23:59.760839 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.760818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2" event={"ID":"5a41def5-0a96-49cb-95fd-fd46b7743ef3","Type":"ContainerStarted","Data":"5fb08607c147e09bea9b6a8394b03875a9717b0d66873f9b307629d432fd1941"}
Sep 15 21:23:59.760935 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.760867    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7863-b9e177" podUID="e0ee9b1a-fa70-4e3d-bcae-a56df780c483" containerName="test-curl-7863-b9e177" containerID="cri-o://4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267" gracePeriod=2
Sep 15 21:23:59.775268 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:23:59.775213    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7863-74252b" podStartSLOduration=1.775198056 podStartE2EDuration="1.775198056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:59.77319842 +0000 UTC m=+829.736255814" watchObservedRunningTime="2026-09-15 21:23:59.775198056 +0000 UTC m=+829.738255492"
Sep 15 21:24:00.401250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.401186    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7863-74252b"]
Sep 15 21:24:00.410104 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.410065    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7863-74252b"]
Sep 15 21:24:00.559228 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.559151    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"]
Sep 15 21:24:00.559595 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.559581    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a312e60-e45a-4cf1-b3d2-77c7e66791eb" containerName="test-curl-7863-74252b"
Sep 15 21:24:00.559647 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.559596    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a312e60-e45a-4cf1-b3d2-77c7e66791eb" containerName="test-curl-7863-74252b"
Sep 15 21:24:00.559712 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.559703    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a312e60-e45a-4cf1-b3d2-77c7e66791eb" containerName="test-curl-7863-74252b"
Sep 15 21:24:00.563004 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.562987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:24:00.565184 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.565162    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-318c22\""
Sep 15 21:24:00.565184 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.565182    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-318c22-dockercfg-hkqkf\""
Sep 15 21:24:00.572856 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.572822    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"]
Sep 15 21:24:00.753157 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.753119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af87cba1-663c-432f-adf7-bf25a39bf502-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp\" (UID: \"af87cba1-663c-432f-adf7-bf25a39bf502\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:24:00.753332 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.753256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfsz\" (UniqueName: \"kubernetes.io/projected/af87cba1-663c-432f-adf7-bf25a39bf502-kube-api-access-plfsz\") pod \"payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp\" (UID: \"af87cba1-663c-432f-adf7-bf25a39bf502\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:24:00.769165 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.769127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2" event={"ID":"5a41def5-0a96-49cb-95fd-fd46b7743ef3","Type":"ContainerStarted","Data":"2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3"}
Sep 15 21:24:00.769628 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.769269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:24:00.769628 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:24:00.769411    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7863-74252b" container="test-curl-7863-74252b"
Sep 15 21:24:00.788459 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.788411    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2" podStartSLOduration=2.361389097 podStartE2EDuration="2.788394473s" totalImagesPullingTime="427.005376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:00.786583478 +0000 UTC m=+830.749640860" watchObservedRunningTime="2026-09-15 21:24:00.788394473 +0000 UTC m=+830.751451853"
Sep 15 21:24:00.854089 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.853995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af87cba1-663c-432f-adf7-bf25a39bf502-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp\" (UID: \"af87cba1-663c-432f-adf7-bf25a39bf502\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:24:00.854271 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.854178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfsz\" (UniqueName: \"kubernetes.io/projected/af87cba1-663c-432f-adf7-bf25a39bf502-kube-api-access-plfsz\") pod \"payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp\" (UID: \"af87cba1-663c-432f-adf7-bf25a39bf502\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:24:00.854854 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.854832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af87cba1-663c-432f-adf7-bf25a39bf502-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp\" (UID: \"af87cba1-663c-432f-adf7-bf25a39bf502\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:24:00.863324 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.863286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfsz\" (UniqueName: \"kubernetes.io/projected/af87cba1-663c-432f-adf7-bf25a39bf502-kube-api-access-plfsz\") pod \"payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp\" (UID: \"af87cba1-663c-432f-adf7-bf25a39bf502\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:24:00.876110 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:00.876078    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:24:01.151561 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:01.151532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"]
Sep 15 21:24:01.152861 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:24:01.152830    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf87cba1_663c_432f_adf7_bf25a39bf502.slice/crio-2242aa094fc80c03fd67ea10de6074cc40728ea14df6528a3aae94f0dc6b047b WatchSource:0}: Error finding container 2242aa094fc80c03fd67ea10de6074cc40728ea14df6528a3aae94f0dc6b047b: Status 404 returned error can't find the container with id 2242aa094fc80c03fd67ea10de6074cc40728ea14df6528a3aae94f0dc6b047b
Sep 15 21:24:01.777146 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:01.777051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp" event={"ID":"af87cba1-663c-432f-adf7-bf25a39bf502","Type":"ContainerStarted","Data":"d5534b205492bc73c6c1195dd1ddce829c70e42e5f59eeccec94f0e43cabb52c"}
Sep 15 21:24:01.777146 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:01.777117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp" event={"ID":"af87cba1-663c-432f-adf7-bf25a39bf502","Type":"ContainerStarted","Data":"2242aa094fc80c03fd67ea10de6074cc40728ea14df6528a3aae94f0dc6b047b"}
Sep 15 21:24:01.777603 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:01.777316    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7863-74252b" podUID="2a312e60-e45a-4cf1-b3d2-77c7e66791eb" containerName="test-curl-7863-74252b" containerID="cri-o://64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3" gracePeriod=2
Sep 15 21:24:01.777603 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:01.777529    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:24:01.796983 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:01.796919    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp" podStartSLOduration=1.7968954510000001 podStartE2EDuration="1.796895451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:01.796326829 +0000 UTC m=+831.759384210" watchObservedRunningTime="2026-09-15 21:24:01.796895451 +0000 UTC m=+831.759952835"
Sep 15 21:24:01.912932 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:01.912905    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-b9e177"
Sep 15 21:24:02.067181 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:02.067096    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5vn8\" (UniqueName: \"kubernetes.io/projected/e0ee9b1a-fa70-4e3d-bcae-a56df780c483-kube-api-access-b5vn8\") pod \"e0ee9b1a-fa70-4e3d-bcae-a56df780c483\" (UID: \"e0ee9b1a-fa70-4e3d-bcae-a56df780c483\") "
Sep 15 21:24:02.069220 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:02.069185    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ee9b1a-fa70-4e3d-bcae-a56df780c483-kube-api-access-b5vn8" (OuterVolumeSpecName: "kube-api-access-b5vn8") pod "e0ee9b1a-fa70-4e3d-bcae-a56df780c483" (UID: "e0ee9b1a-fa70-4e3d-bcae-a56df780c483"). InnerVolumeSpecName "kube-api-access-b5vn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:02.168893 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:02.168843    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5vn8\" (UniqueName: \"kubernetes.io/projected/e0ee9b1a-fa70-4e3d-bcae-a56df780c483-kube-api-access-b5vn8\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:24:02.526952 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:02.526918    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0ee9b1a-fa70-4e3d-bcae-a56df780c483" path="/var/lib/kubelet/pods/e0ee9b1a-fa70-4e3d-bcae-a56df780c483/volumes"
Sep 15 21:24:02.784280 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:02.784184    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0ee9b1a-fa70-4e3d-bcae-a56df780c483" containerID="4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267" exitCode=137
Sep 15 21:24:02.784280 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:02.784223    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-b9e177"
Sep 15 21:24:02.784280 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:02.784265    2563 scope.go:117] "RemoveContainer" containerID="4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267"
Sep 15 21:24:02.793552 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:02.793534    2563 scope.go:117] "RemoveContainer" containerID="4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267"
Sep 15 21:24:02.793868 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:24:02.793847    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267\": container with ID starting with 4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267 not found: ID does not exist" containerID="4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267"
Sep 15 21:24:02.793935 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:02.793880    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267"} err="failed to get container status \"4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267\": rpc error: code = NotFound desc = could not find container \"4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267\": container with ID starting with 4474ecb2301a65e6d0790005099ffc3a78253ad91283af0c18a7606a507ad267 not found: ID does not exist"
Sep 15 21:24:03.973771 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:03.973743    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-74252b"
Sep 15 21:24:03.981689 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:03.981658    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djbpv\" (UniqueName: \"kubernetes.io/projected/2a312e60-e45a-4cf1-b3d2-77c7e66791eb-kube-api-access-djbpv\") pod \"2a312e60-e45a-4cf1-b3d2-77c7e66791eb\" (UID: \"2a312e60-e45a-4cf1-b3d2-77c7e66791eb\") "
Sep 15 21:24:03.984053 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:03.984019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a312e60-e45a-4cf1-b3d2-77c7e66791eb-kube-api-access-djbpv" (OuterVolumeSpecName: "kube-api-access-djbpv") pod "2a312e60-e45a-4cf1-b3d2-77c7e66791eb" (UID: "2a312e60-e45a-4cf1-b3d2-77c7e66791eb"). InnerVolumeSpecName "kube-api-access-djbpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:04.082643 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:04.082562    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djbpv\" (UniqueName: \"kubernetes.io/projected/2a312e60-e45a-4cf1-b3d2-77c7e66791eb-kube-api-access-djbpv\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:24:04.527737 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:04.527698    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a312e60-e45a-4cf1-b3d2-77c7e66791eb" path="/var/lib/kubelet/pods/2a312e60-e45a-4cf1-b3d2-77c7e66791eb/volumes"
Sep 15 21:24:04.802390 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:04.802275    2563 generic.go:356] "Generic (PLEG): container finished" podID="2a312e60-e45a-4cf1-b3d2-77c7e66791eb" containerID="64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3" exitCode=137
Sep 15 21:24:04.802390 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:04.802322    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-74252b"
Sep 15 21:24:04.802390 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:04.802329    2563 scope.go:117] "RemoveContainer" containerID="64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3"
Sep 15 21:24:04.821941 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:04.821885    2563 scope.go:117] "RemoveContainer" containerID="64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3"
Sep 15 21:24:04.822268 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:24:04.822242    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3\": container with ID starting with 64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3 not found: ID does not exist" containerID="64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3"
Sep 15 21:24:04.822369 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:04.822280    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3"} err="failed to get container status \"64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3\": rpc error: code = NotFound desc = could not find container \"64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3\": container with ID starting with 64efb1e2bf5b1389576f1bf5a1ac8472f2a512ae297872cb9c3674e011ad5ea3 not found: ID does not exist"
Sep 15 21:24:06.783300 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:06.783270    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:24:09.284595 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.284560    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"]
Sep 15 21:24:09.287939 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.287923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:09.290132 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.290112    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d0431a5e\""
Sep 15 21:24:09.298901 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.298872    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"]
Sep 15 21:24:09.312665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.312637    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"]
Sep 15 21:24:09.316121 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.316102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:09.318292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.318273    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d0431a5e-dockercfg-wv48j\""
Sep 15 21:24:09.323772 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.323749    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"]
Sep 15 21:24:09.329995 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.329975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwr2t\" (UniqueName: \"kubernetes.io/projected/5610ea5e-e51c-49b1-8927-abc282a1bc8a-kube-api-access-jwr2t\") pod \"payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6\" (UID: \"5610ea5e-e51c-49b1-8927-abc282a1bc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:09.330126 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.330103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96zq\" (UniqueName: \"kubernetes.io/projected/500f6074-1240-44f6-ab3f-699470f9fa26-kube-api-access-l96zq\") pod \"payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k\" (UID: \"500f6074-1240-44f6-ab3f-699470f9fa26\") " pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:09.330200 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.330154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5610ea5e-e51c-49b1-8927-abc282a1bc8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6\" (UID: \"5610ea5e-e51c-49b1-8927-abc282a1bc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:09.330253 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.330228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500f6074-1240-44f6-ab3f-699470f9fa26-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k\" (UID: \"500f6074-1240-44f6-ab3f-699470f9fa26\") " pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:09.431064 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.431029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500f6074-1240-44f6-ab3f-699470f9fa26-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k\" (UID: \"500f6074-1240-44f6-ab3f-699470f9fa26\") " pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:09.431217 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.431073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwr2t\" (UniqueName: \"kubernetes.io/projected/5610ea5e-e51c-49b1-8927-abc282a1bc8a-kube-api-access-jwr2t\") pod \"payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6\" (UID: \"5610ea5e-e51c-49b1-8927-abc282a1bc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:09.431217 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.431187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96zq\" (UniqueName: \"kubernetes.io/projected/500f6074-1240-44f6-ab3f-699470f9fa26-kube-api-access-l96zq\") pod \"payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k\" (UID: \"500f6074-1240-44f6-ab3f-699470f9fa26\") " pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:09.431217 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.431207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5610ea5e-e51c-49b1-8927-abc282a1bc8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6\" (UID: \"5610ea5e-e51c-49b1-8927-abc282a1bc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:09.431970 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.431948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5610ea5e-e51c-49b1-8927-abc282a1bc8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6\" (UID: \"5610ea5e-e51c-49b1-8927-abc282a1bc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:09.432102 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.432084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500f6074-1240-44f6-ab3f-699470f9fa26-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k\" (UID: \"500f6074-1240-44f6-ab3f-699470f9fa26\") " pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:09.440248 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.440224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwr2t\" (UniqueName: \"kubernetes.io/projected/5610ea5e-e51c-49b1-8927-abc282a1bc8a-kube-api-access-jwr2t\") pod \"payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6\" (UID: \"5610ea5e-e51c-49b1-8927-abc282a1bc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:09.440368 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.440333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96zq\" (UniqueName: \"kubernetes.io/projected/500f6074-1240-44f6-ab3f-699470f9fa26-kube-api-access-l96zq\") pod \"payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k\" (UID: \"500f6074-1240-44f6-ab3f-699470f9fa26\") " pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:09.600883 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.600797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:09.629340 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.629309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:09.884280 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.884259    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"]
Sep 15 21:24:09.885113 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:24:09.885089    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5610ea5e_e51c_49b1_8927_abc282a1bc8a.slice/crio-fd2da611f7101c43d25f6882c81f26ed45e9f51e0f8c71d4ed3db81f5e6816f4 WatchSource:0}: Error finding container fd2da611f7101c43d25f6882c81f26ed45e9f51e0f8c71d4ed3db81f5e6816f4: Status 404 returned error can't find the container with id fd2da611f7101c43d25f6882c81f26ed45e9f51e0f8c71d4ed3db81f5e6816f4
Sep 15 21:24:09.911133 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:09.911111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"]
Sep 15 21:24:09.920583 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:24:09.920559    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500f6074_1240_44f6_ab3f_699470f9fa26.slice/crio-1920ddd4088ec2f59060705f19dc7d8d2e13bc4a2159883ccb02acd9ce61b606 WatchSource:0}: Error finding container 1920ddd4088ec2f59060705f19dc7d8d2e13bc4a2159883ccb02acd9ce61b606: Status 404 returned error can't find the container with id 1920ddd4088ec2f59060705f19dc7d8d2e13bc4a2159883ccb02acd9ce61b606
Sep 15 21:24:10.285481 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.285396    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"]
Sep 15 21:24:10.290245 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.290217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:24:10.292659 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.292630    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-04c28d\""
Sep 15 21:24:10.292776 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.292681    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-04c28d-dockercfg-fwrtn\""
Sep 15 21:24:10.299624 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.299235    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"]
Sep 15 21:24:10.340806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.340778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2j56\" (UniqueName: \"kubernetes.io/projected/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-kube-api-access-z2j56\") pod \"payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s\" (UID: \"2e1725cb-3b3e-4613-a67c-d1ec692d1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:24:10.340984 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.340815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s\" (UID: \"2e1725cb-3b3e-4613-a67c-d1ec692d1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:24:10.442599 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.442559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2j56\" (UniqueName: \"kubernetes.io/projected/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-kube-api-access-z2j56\") pod \"payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s\" (UID: \"2e1725cb-3b3e-4613-a67c-d1ec692d1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:24:10.442797 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.442607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s\" (UID: \"2e1725cb-3b3e-4613-a67c-d1ec692d1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:24:10.446420 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.446376    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-04c28d\""
Sep 15 21:24:10.454395 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.454365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s\" (UID: \"2e1725cb-3b3e-4613-a67c-d1ec692d1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:24:10.457121 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.457088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2j56\" (UniqueName: \"kubernetes.io/projected/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-kube-api-access-z2j56\") pod \"payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s\" (UID: \"2e1725cb-3b3e-4613-a67c-d1ec692d1f51\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:24:10.606405 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.606376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-04c28d-dockercfg-fwrtn\""
Sep 15 21:24:10.615091 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.615063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:24:10.665483 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.665451    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"]
Sep 15 21:24:10.670901 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.670876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.674094 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.673466    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:24:10.674094 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.673862    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f457e5-openshift-default-dockercfg-lgtfc\""
Sep 15 21:24:10.685243 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.685218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"]
Sep 15 21:24:10.744443 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.744410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-podinfo\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.744602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.744452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istiod-ca-cert\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.744602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.744500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-certs\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.744602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.744518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-envoy\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.744794 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.744609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-socket\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.744794 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.744645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjtmx\" (UniqueName: \"kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-kube-api-access-wjtmx\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.744794 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.744680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-token\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.744905 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.744801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-credential-socket\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.744905 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.744858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-data\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.845506 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-data\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.845506 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-podinfo\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.845506 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k" event={"ID":"500f6074-1240-44f6-ab3f-699470f9fa26","Type":"ContainerStarted","Data":"942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f"}
Sep 15 21:24:10.845506 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istiod-ca-cert\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.845833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-certs\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.845833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-envoy\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.845833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-socket\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.845833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjtmx\" (UniqueName: \"kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-kube-api-access-wjtmx\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.845833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-token\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.845833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-credential-socket\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.846107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-data\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.846107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-certs\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.846107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.845537    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:10.846107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.846012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k" event={"ID":"500f6074-1240-44f6-ab3f-699470f9fa26","Type":"ContainerStarted","Data":"1920ddd4088ec2f59060705f19dc7d8d2e13bc4a2159883ccb02acd9ce61b606"}
Sep 15 21:24:10.846304 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.846226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-socket\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.846426 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.846400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-credential-socket\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.846663 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.846640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istiod-ca-cert\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.847878 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.847854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6" event={"ID":"5610ea5e-e51c-49b1-8927-abc282a1bc8a","Type":"ContainerStarted","Data":"3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377"}
Sep 15 21:24:10.847993 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.847900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6" event={"ID":"5610ea5e-e51c-49b1-8927-abc282a1bc8a","Type":"ContainerStarted","Data":"fd2da611f7101c43d25f6882c81f26ed45e9f51e0f8c71d4ed3db81f5e6816f4"}
Sep 15 21:24:10.847993 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.847987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:10.848117 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.848098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-envoy\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.848267 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.848249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-podinfo\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.854301 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.854274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-token\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.854425 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.854363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjtmx\" (UniqueName: \"kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-kube-api-access-wjtmx\") pod \"e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") " pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:10.865163 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.865115    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k" podStartSLOduration=1.865097674 podStartE2EDuration="1.865097674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:10.861071799 +0000 UTC m=+840.824129204" watchObservedRunningTime="2026-09-15 21:24:10.865097674 +0000 UTC m=+840.828155055"
Sep 15 21:24:10.878693 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.878648    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6" podStartSLOduration=1.878635891 podStartE2EDuration="1.878635891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:10.875731325 +0000 UTC m=+840.838788706" watchObservedRunningTime="2026-09-15 21:24:10.878635891 +0000 UTC m=+840.841693271"
Sep 15 21:24:10.904138 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.904113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"]
Sep 15 21:24:10.905111 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:24:10.905089    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1725cb_3b3e_4613_a67c_d1ec692d1f51.slice/crio-2e3f6142a041fcde2888ecb7fc014208d86b35779d63871bf19db03f00da43d9 WatchSource:0}: Error finding container 2e3f6142a041fcde2888ecb7fc014208d86b35779d63871bf19db03f00da43d9: Status 404 returned error can't find the container with id 2e3f6142a041fcde2888ecb7fc014208d86b35779d63871bf19db03f00da43d9
Sep 15 21:24:10.990769 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:10.990737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:11.276717 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:11.276690    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"]
Sep 15 21:24:11.277650 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:24:11.277598    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375b66c1_4e10_4dc4_8315_3ec6be3868a2.slice/crio-7d1243c44c02b5722fd6b8b36c6efc7515c62ed1087db4ebea0d6080b2b39e90 WatchSource:0}: Error finding container 7d1243c44c02b5722fd6b8b36c6efc7515c62ed1087db4ebea0d6080b2b39e90: Status 404 returned error can't find the container with id 7d1243c44c02b5722fd6b8b36c6efc7515c62ed1087db4ebea0d6080b2b39e90
Sep 15 21:24:11.859983 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:11.859911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s" event={"ID":"2e1725cb-3b3e-4613-a67c-d1ec692d1f51","Type":"ContainerStarted","Data":"fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f"}
Sep 15 21:24:11.860469 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:11.860028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s" event={"ID":"2e1725cb-3b3e-4613-a67c-d1ec692d1f51","Type":"ContainerStarted","Data":"2e3f6142a041fcde2888ecb7fc014208d86b35779d63871bf19db03f00da43d9"}
Sep 15 21:24:11.860469 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:11.860078    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:24:11.861812 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:11.861782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" event={"ID":"375b66c1-4e10-4dc4-8315-3ec6be3868a2","Type":"ContainerStarted","Data":"7d1243c44c02b5722fd6b8b36c6efc7515c62ed1087db4ebea0d6080b2b39e90"}
Sep 15 21:24:11.884067 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:11.884014    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s" podStartSLOduration=1.8839986180000001 podStartE2EDuration="1.883998618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:11.883705294 +0000 UTC m=+841.846762675" watchObservedRunningTime="2026-09-15 21:24:11.883998618 +0000 UTC m=+841.847055999"
Sep 15 21:24:13.705109 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:13.705066    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:24:13.705413 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:13.705139    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:24:13.705413 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:13.705168    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:24:13.796031 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:13.796010    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:24:13.881236 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:13.881198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" event={"ID":"375b66c1-4e10-4dc4-8315-3ec6be3868a2","Type":"ContainerStarted","Data":"e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6"}
Sep 15 21:24:13.901696 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:13.901631    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" podStartSLOduration=1.4770663050000001 podStartE2EDuration="3.90160881s" totalImagesPullingTime="2.424542505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:13.89996708 +0000 UTC m=+843.863024461" watchObservedRunningTime="2026-09-15 21:24:13.90160881 +0000 UTC m=+843.864666192"
Sep 15 21:24:13.991811 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:13.991733    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:14.997486 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:14.997459    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:15.894770 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:15.894735    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:15.895656 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:15.895632    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:21.864300 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:21.864271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:22.872334 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:22.872306    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:24:22.872761 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:22.872742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:36.505558 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:36.505523    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"]
Sep 15 21:24:36.507012 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:36.506940    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k" podUID="500f6074-1240-44f6-ab3f-699470f9fa26" containerName="payload-processing" containerID="cri-o://942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f" gracePeriod=30
Sep 15 21:24:36.512618 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:36.512408    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"]
Sep 15 21:24:36.512907 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:36.512618    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6" podUID="5610ea5e-e51c-49b1-8927-abc282a1bc8a" containerName="payload-pre-processing" containerID="cri-o://3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377" gracePeriod=30
Sep 15 21:24:36.867391 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:36.865264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:36.883405 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:36.883327    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:37.024106 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.024012    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwr2t\" (UniqueName: \"kubernetes.io/projected/5610ea5e-e51c-49b1-8927-abc282a1bc8a-kube-api-access-jwr2t\") pod \"5610ea5e-e51c-49b1-8927-abc282a1bc8a\" (UID: \"5610ea5e-e51c-49b1-8927-abc282a1bc8a\") "
Sep 15 21:24:37.024106 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.024057    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5610ea5e-e51c-49b1-8927-abc282a1bc8a-plugins-config-volume\") pod \"5610ea5e-e51c-49b1-8927-abc282a1bc8a\" (UID: \"5610ea5e-e51c-49b1-8927-abc282a1bc8a\") "
Sep 15 21:24:37.024330 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.024121    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500f6074-1240-44f6-ab3f-699470f9fa26-plugins-config-volume\") pod \"500f6074-1240-44f6-ab3f-699470f9fa26\" (UID: \"500f6074-1240-44f6-ab3f-699470f9fa26\") "
Sep 15 21:24:37.024330 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.024164    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l96zq\" (UniqueName: \"kubernetes.io/projected/500f6074-1240-44f6-ab3f-699470f9fa26-kube-api-access-l96zq\") pod \"500f6074-1240-44f6-ab3f-699470f9fa26\" (UID: \"500f6074-1240-44f6-ab3f-699470f9fa26\") "
Sep 15 21:24:37.024503 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.024478    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5610ea5e-e51c-49b1-8927-abc282a1bc8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5610ea5e-e51c-49b1-8927-abc282a1bc8a" (UID: "5610ea5e-e51c-49b1-8927-abc282a1bc8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:24:37.024567 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.024517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500f6074-1240-44f6-ab3f-699470f9fa26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "500f6074-1240-44f6-ab3f-699470f9fa26" (UID: "500f6074-1240-44f6-ab3f-699470f9fa26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:24:37.024665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.024646    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5610ea5e-e51c-49b1-8927-abc282a1bc8a-plugins-config-volume\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:24:37.024724 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.024670    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500f6074-1240-44f6-ab3f-699470f9fa26-plugins-config-volume\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:24:37.026298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.026275    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500f6074-1240-44f6-ab3f-699470f9fa26-kube-api-access-l96zq" (OuterVolumeSpecName: "kube-api-access-l96zq") pod "500f6074-1240-44f6-ab3f-699470f9fa26" (UID: "500f6074-1240-44f6-ab3f-699470f9fa26"). InnerVolumeSpecName "kube-api-access-l96zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:37.026490 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.026475    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5610ea5e-e51c-49b1-8927-abc282a1bc8a-kube-api-access-jwr2t" (OuterVolumeSpecName: "kube-api-access-jwr2t") pod "5610ea5e-e51c-49b1-8927-abc282a1bc8a" (UID: "5610ea5e-e51c-49b1-8927-abc282a1bc8a"). InnerVolumeSpecName "kube-api-access-jwr2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:37.031428 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.031406    2563 generic.go:356] "Generic (PLEG): container finished" podID="5610ea5e-e51c-49b1-8927-abc282a1bc8a" containerID="3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377" exitCode=0
Sep 15 21:24:37.031523 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.031447    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"
Sep 15 21:24:37.031523 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.031464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6" event={"ID":"5610ea5e-e51c-49b1-8927-abc282a1bc8a","Type":"ContainerDied","Data":"3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377"}
Sep 15 21:24:37.031637 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.031526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6" event={"ID":"5610ea5e-e51c-49b1-8927-abc282a1bc8a","Type":"ContainerDied","Data":"fd2da611f7101c43d25f6882c81f26ed45e9f51e0f8c71d4ed3db81f5e6816f4"}
Sep 15 21:24:37.031637 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.031558    2563 scope.go:117] "RemoveContainer" containerID="3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377"
Sep 15 21:24:37.032968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.032936    2563 generic.go:356] "Generic (PLEG): container finished" podID="500f6074-1240-44f6-ab3f-699470f9fa26" containerID="942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f" exitCode=0
Sep 15 21:24:37.033054 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.032972    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"
Sep 15 21:24:37.033054 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.032985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k" event={"ID":"500f6074-1240-44f6-ab3f-699470f9fa26","Type":"ContainerDied","Data":"942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f"}
Sep 15 21:24:37.033054 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.033022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k" event={"ID":"500f6074-1240-44f6-ab3f-699470f9fa26","Type":"ContainerDied","Data":"1920ddd4088ec2f59060705f19dc7d8d2e13bc4a2159883ccb02acd9ce61b606"}
Sep 15 21:24:37.041962 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.041941    2563 scope.go:117] "RemoveContainer" containerID="3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377"
Sep 15 21:24:37.042243 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:24:37.042222    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377\": container with ID starting with 3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377 not found: ID does not exist" containerID="3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377"
Sep 15 21:24:37.042331 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.042250    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377"} err="failed to get container status \"3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377\": rpc error: code = NotFound desc = could not find container \"3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377\": container with ID starting with 3f0b4a03ae3bf65d27cb3ceee63f1d136bd6bd269bd0be727b5c7f4bebf09377 not found: ID does not exist"
Sep 15 21:24:37.042331 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.042265    2563 scope.go:117] "RemoveContainer" containerID="942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f"
Sep 15 21:24:37.070478 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.068470    2563 scope.go:117] "RemoveContainer" containerID="942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f"
Sep 15 21:24:37.075683 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:24:37.075500    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f\": container with ID starting with 942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f not found: ID does not exist" containerID="942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f"
Sep 15 21:24:37.075683 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.075572    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f"} err="failed to get container status \"942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f\": rpc error: code = NotFound desc = could not find container \"942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f\": container with ID starting with 942335056a15c46618cf5fc511208c5ec4879c1f0c066214b8952fa89dd1ca5f not found: ID does not exist"
Sep 15 21:24:37.075683 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.075616    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"]
Sep 15 21:24:37.076998 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.076973    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0431a5e-5997b55dd5-5hf4k"]
Sep 15 21:24:37.082959 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.082938    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"]
Sep 15 21:24:37.086486 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.086462    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0431a5e-5b986769c8-7nrg6"]
Sep 15 21:24:37.126162 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.126138    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwr2t\" (UniqueName: \"kubernetes.io/projected/5610ea5e-e51c-49b1-8927-abc282a1bc8a-kube-api-access-jwr2t\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:24:37.126162 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:37.126160    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l96zq\" (UniqueName: \"kubernetes.io/projected/500f6074-1240-44f6-ab3f-699470f9fa26-kube-api-access-l96zq\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:24:38.529111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:38.529073    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="500f6074-1240-44f6-ab3f-699470f9fa26" path="/var/lib/kubelet/pods/500f6074-1240-44f6-ab3f-699470f9fa26/volumes"
Sep 15 21:24:38.530187 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:38.530163    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5610ea5e-e51c-49b1-8927-abc282a1bc8a" path="/var/lib/kubelet/pods/5610ea5e-e51c-49b1-8927-abc282a1bc8a/volumes"
Sep 15 21:24:42.069669 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.069591    2563 generic.go:356] "Generic (PLEG): container finished" podID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerID="e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6" exitCode=0
Sep 15 21:24:42.070064 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.069663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" event={"ID":"375b66c1-4e10-4dc4-8315-3ec6be3868a2","Type":"ContainerDied","Data":"e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6"}
Sep 15 21:24:42.070110 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.070072    2563 scope.go:117] "RemoveContainer" containerID="e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6"
Sep 15 21:24:42.071227 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.071201    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:24:42.071278 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.071268    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:24:42.071312 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.071299    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:24:42.440815 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.440763    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"]
Sep 15 21:24:42.441518 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.441474    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5610ea5e-e51c-49b1-8927-abc282a1bc8a" containerName="payload-pre-processing"
Sep 15 21:24:42.441518 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.441497    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5610ea5e-e51c-49b1-8927-abc282a1bc8a" containerName="payload-pre-processing"
Sep 15 21:24:42.441748 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.441546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="500f6074-1240-44f6-ab3f-699470f9fa26" containerName="payload-processing"
Sep 15 21:24:42.441748 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.441556    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="500f6074-1240-44f6-ab3f-699470f9fa26" containerName="payload-processing"
Sep 15 21:24:42.441748 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.441682    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5610ea5e-e51c-49b1-8927-abc282a1bc8a" containerName="payload-pre-processing"
Sep 15 21:24:42.441748 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.441693    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="500f6074-1240-44f6-ab3f-699470f9fa26" containerName="payload-processing"
Sep 15 21:24:42.443880 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.443860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.445813 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.445793    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-edbfd1c5-serving-cert\""
Sep 15 21:24:42.455190 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.455090    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"]
Sep 15 21:24:42.482337 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.482305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-maas-api-tls\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.482337 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.482338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-openshift-service-ca\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.482596 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.482415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-metrics-certs\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.482596 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.482448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqq5h\" (UniqueName: \"kubernetes.io/projected/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-kube-api-access-mqq5h\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.583383 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.583336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-maas-api-tls\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.583383 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.583383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-openshift-service-ca\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.583601 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.583418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-metrics-certs\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.583601 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:24:42.583435    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-serving-cert: secret "maas-api-e2e-migrate-edbfd1c5-serving-cert" not found
Sep 15 21:24:42.583601 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.583447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqq5h\" (UniqueName: \"kubernetes.io/projected/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-kube-api-access-mqq5h\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.583601 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:24:42.583493    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-maas-api-tls podName:94ed7df8-2794-4b08-a6a1-22d6d97eb3ed nodeName:}" failed. No retries permitted until 2026-09-15 21:24:43.083476709 +0000 UTC m=+873.046534072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-maas-api-tls") pod "maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5" (UID: "94ed7df8-2794-4b08-a6a1-22d6d97eb3ed") : secret "maas-api-e2e-migrate-edbfd1c5-serving-cert" not found
Sep 15 21:24:42.584105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.584088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-openshift-service-ca\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.585857 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.585841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-metrics-certs\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:42.594388 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:42.594362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqq5h\" (UniqueName: \"kubernetes.io/projected/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-kube-api-access-mqq5h\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:43.078413 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:43.078375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" event={"ID":"375b66c1-4e10-4dc4-8315-3ec6be3868a2","Type":"ContainerStarted","Data":"86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a"}
Sep 15 21:24:43.087443 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:43.087409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-maas-api-tls\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:43.090408 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:43.090383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-maas-api-tls\") pod \"maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:43.358592 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:43.358488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:43.637191 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:43.637165    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"]
Sep 15 21:24:43.638412 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:24:43.638392    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ed7df8_2794_4b08_a6a1_22d6d97eb3ed.slice/crio-28689cd07b0f365e39156a26692f0c99944e4ceb3bf98df982f15a1a2bf6d8ca WatchSource:0}: Error finding container 28689cd07b0f365e39156a26692f0c99944e4ceb3bf98df982f15a1a2bf6d8ca: Status 404 returned error can't find the container with id 28689cd07b0f365e39156a26692f0c99944e4ceb3bf98df982f15a1a2bf6d8ca
Sep 15 21:24:43.991703 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:43.991609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:43.992673 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:43.992634    2563 patch_prober.go:28] interesting pod/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 21:24:43.992802 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:43.992705    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 21:24:44.085758 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:44.085735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5" event={"ID":"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed","Type":"ContainerStarted","Data":"28689cd07b0f365e39156a26692f0c99944e4ceb3bf98df982f15a1a2bf6d8ca"}
Sep 15 21:24:44.991505 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:44.991469    2563 patch_prober.go:28] interesting pod/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 21:24:44.991683 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:44.991525    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 21:24:45.093202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:45.093163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5" event={"ID":"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed","Type":"ContainerStarted","Data":"fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d"}
Sep 15 21:24:45.093673 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:45.093286    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:24:45.112292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:45.112243    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5" podStartSLOduration=2.673125207 podStartE2EDuration="3.112228365s" totalImagesPullingTime="439.103158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:45.110698353 +0000 UTC m=+875.073755736" watchObservedRunningTime="2026-09-15 21:24:45.112228365 +0000 UTC m=+875.075285746"
Sep 15 21:24:45.894907 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:45.894871    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:45.991660 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:45.991626    2563 patch_prober.go:28] interesting pod/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 21:24:45.991827 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:45.991678    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 21:24:46.991580 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:46.991545    2563 patch_prober.go:28] interesting pod/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 21:24:46.991968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:46.991622    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 21:24:47.991534 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:47.991497    2563 patch_prober.go:28] interesting pod/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 21:24:47.991710 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:47.991558    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 21:24:48.995101 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:48.995068    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:48.995952 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:48.995933    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:24:51.104976 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:24:51.104948    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:25:02.992486 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:02.992413    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"]
Sep 15 21:25:02.996472 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:02.996451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:02.998527 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:02.998510    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-318c22\"/\"kube-root-ca.crt\""
Sep 15 21:25:02.999533 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:02.999511    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-318c22\"/\"openshift-service-ca.crt\""
Sep 15 21:25:02.999533 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:02.999523    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-318c22\"/\"default-dockercfg-7tnmn\""
Sep 15 21:25:02.999710 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:02.999559    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-318c22\"/\"simulator-w0-318c22-kserve-self-signed-certs\""
Sep 15 21:25:03.003797 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.003772    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"]
Sep 15 21:25:03.078431 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.078408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tls-certs\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.078536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.078445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-model-cache\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.078536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.078481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tmp-dir\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.078608 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.078535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6s58\" (UniqueName: \"kubernetes.io/projected/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-kube-api-access-z6s58\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.078608 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.078575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-dshm\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.078608 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.078604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-home\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.179171 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.179143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6s58\" (UniqueName: \"kubernetes.io/projected/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-kube-api-access-z6s58\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.179375 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.179194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-dshm\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.179433 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.179393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-home\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.179519 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.179503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tls-certs\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.179562 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.179549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-model-cache\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.179613 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.179598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tmp-dir\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.179789 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.179767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-home\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.179789 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.179784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-model-cache\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.179915 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.179847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tmp-dir\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.181416 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.181395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-dshm\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.181743 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.181723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tls-certs\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.189946 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.189927    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6s58\" (UniqueName: \"kubernetes.io/projected/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-kube-api-access-z6s58\") pod \"simulator-w0-318c22-kserve-6fb4f8699d-8twf7\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") " pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.324291 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.324207    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:03.600283 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:03.600255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"]
Sep 15 21:25:03.602198 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:25:03.602174    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6ff16e5_0dcb_4bd0_aa3d_0661c36746aa.slice/crio-e99f7ff939229d1e30d07d385a6b98b1e7a814de8b03633167ed714b010dca79 WatchSource:0}: Error finding container e99f7ff939229d1e30d07d385a6b98b1e7a814de8b03633167ed714b010dca79: Status 404 returned error can't find the container with id e99f7ff939229d1e30d07d385a6b98b1e7a814de8b03633167ed714b010dca79
Sep 15 21:25:04.212944 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:04.212902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7" event={"ID":"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa","Type":"ContainerStarted","Data":"1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0"}
Sep 15 21:25:04.213377 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:04.212955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7" event={"ID":"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa","Type":"ContainerStarted","Data":"e99f7ff939229d1e30d07d385a6b98b1e7a814de8b03633167ed714b010dca79"}
Sep 15 21:25:04.229641 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:04.229596    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7" podStartSLOduration=2.229582048 podStartE2EDuration="2.229582048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:04.228371556 +0000 UTC m=+894.191428931" watchObservedRunningTime="2026-09-15 21:25:04.229582048 +0000 UTC m=+894.192639428"
Sep 15 21:25:11.694531 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.694498    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"]
Sep 15 21:25:11.699731 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.699707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.701788 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.701761    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"kube-root-ca.crt\""
Sep 15 21:25:11.702503 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.702474    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"simulator-w4-6f66f7-kserve-self-signed-certs\""
Sep 15 21:25:11.702695 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.702675    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"default-dockercfg-vlvng\""
Sep 15 21:25:11.702886 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.702867    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"openshift-service-ca.crt\""
Sep 15 21:25:11.705778 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.705756    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"]
Sep 15 21:25:11.776488 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.776452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tls-certs\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.776488 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.776488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-home\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.776725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.776531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-model-cache\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.776725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.776621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8zl\" (UniqueName: \"kubernetes.io/projected/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-kube-api-access-jp8zl\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.776725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.776651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tmp-dir\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.776725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.776706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-dshm\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.877920 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.877886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-model-cache\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.878130 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.877952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8zl\" (UniqueName: \"kubernetes.io/projected/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-kube-api-access-jp8zl\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.878130 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.877986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tmp-dir\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.878130 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.878028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-dshm\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.878130 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.878132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tls-certs\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.878397 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.878154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-home\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.878397 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.878288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-model-cache\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.878526 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.878500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-home\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.878660 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.878645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tmp-dir\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.880511 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.880484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-dshm\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.880914 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.880894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tls-certs\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:11.886054 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:11.886031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8zl\" (UniqueName: \"kubernetes.io/projected/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-kube-api-access-jp8zl\") pod \"simulator-w4-6f66f7-kserve-786c449848-p2qkv\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") " pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:12.011247 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.011159    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:12.293135 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.293105    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"]
Sep 15 21:25:12.294730 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:25:12.294690    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e32eff9_bb23_4daa_8cc3_f5e3ae09488d.slice/crio-da333b4fd199dd28faceb2cb0fc1dd5b15ddf2445fe6212d5f67e8782353e639 WatchSource:0}: Error finding container da333b4fd199dd28faceb2cb0fc1dd5b15ddf2445fe6212d5f67e8782353e639: Status 404 returned error can't find the container with id da333b4fd199dd28faceb2cb0fc1dd5b15ddf2445fe6212d5f67e8782353e639
Sep 15 21:25:12.575099 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.575018    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"]
Sep 15 21:25:12.575259 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.575220    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5" podUID="94ed7df8-2794-4b08-a6a1-22d6d97eb3ed" containerName="maas-api" containerID="cri-o://fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d" gracePeriod=30
Sep 15 21:25:12.814691 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.814664    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:25:12.989687 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.989652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-maas-api-tls\") pod \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") "
Sep 15 21:25:12.989861 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.989761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqq5h\" (UniqueName: \"kubernetes.io/projected/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-kube-api-access-mqq5h\") pod \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") "
Sep 15 21:25:12.989861 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.989828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-openshift-service-ca\") pod \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") "
Sep 15 21:25:12.989946 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.989864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-metrics-certs\") pod \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\" (UID: \"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed\") "
Sep 15 21:25:12.990224 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.990197    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94ed7df8-2794-4b08-a6a1-22d6d97eb3ed" (UID: "94ed7df8-2794-4b08-a6a1-22d6d97eb3ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:25:12.991835 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.991811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94ed7df8-2794-4b08-a6a1-22d6d97eb3ed" (UID: "94ed7df8-2794-4b08-a6a1-22d6d97eb3ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:25:12.991918 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.991860    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-kube-api-access-mqq5h" (OuterVolumeSpecName: "kube-api-access-mqq5h") pod "94ed7df8-2794-4b08-a6a1-22d6d97eb3ed" (UID: "94ed7df8-2794-4b08-a6a1-22d6d97eb3ed"). InnerVolumeSpecName "kube-api-access-mqq5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:12.991918 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:12.991888    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94ed7df8-2794-4b08-a6a1-22d6d97eb3ed" (UID: "94ed7df8-2794-4b08-a6a1-22d6d97eb3ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:25:13.090892 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.090859    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqq5h\" (UniqueName: \"kubernetes.io/projected/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-kube-api-access-mqq5h\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:25:13.090892 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.090887    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-openshift-service-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:25:13.090892 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.090897    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-metrics-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:25:13.091130 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.090905    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed-maas-api-tls\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:25:13.274846 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.274754    2563 generic.go:356] "Generic (PLEG): container finished" podID="94ed7df8-2794-4b08-a6a1-22d6d97eb3ed" containerID="fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d" exitCode=0
Sep 15 21:25:13.274846 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.274803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"
Sep 15 21:25:13.274846 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.274828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5" event={"ID":"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed","Type":"ContainerDied","Data":"fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d"}
Sep 15 21:25:13.275113 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.274865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5" event={"ID":"94ed7df8-2794-4b08-a6a1-22d6d97eb3ed","Type":"ContainerDied","Data":"28689cd07b0f365e39156a26692f0c99944e4ceb3bf98df982f15a1a2bf6d8ca"}
Sep 15 21:25:13.275113 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.274885    2563 scope.go:117] "RemoveContainer" containerID="fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d"
Sep 15 21:25:13.276836 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.276808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv" event={"ID":"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d","Type":"ContainerStarted","Data":"2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b"}
Sep 15 21:25:13.276935 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.276852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv" event={"ID":"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d","Type":"ContainerStarted","Data":"da333b4fd199dd28faceb2cb0fc1dd5b15ddf2445fe6212d5f67e8782353e639"}
Sep 15 21:25:13.285438 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.285422    2563 scope.go:117] "RemoveContainer" containerID="fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d"
Sep 15 21:25:13.285689 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:25:13.285673    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d\": container with ID starting with fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d not found: ID does not exist" containerID="fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d"
Sep 15 21:25:13.285750 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.285697    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d"} err="failed to get container status \"fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d\": rpc error: code = NotFound desc = could not find container \"fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d\": container with ID starting with fa0fabd787397db663112ec2e04adeae2c482514177430bf8e355f359ecc799d not found: ID does not exist"
Sep 15 21:25:13.300785 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.300724    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv" podStartSLOduration=2.300703794 podStartE2EDuration="2.300703794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:13.296506831 +0000 UTC m=+903.259564214" watchObservedRunningTime="2026-09-15 21:25:13.300703794 +0000 UTC m=+903.263761178"
Sep 15 21:25:13.315503 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.315473    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"]
Sep 15 21:25:13.319633 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.319604    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-edbfd1c5-5557bb7f75-fl4s5"]
Sep 15 21:25:13.325005 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.324973    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:13.325366 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.325333    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:13.337807 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:13.337787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:14.296008 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:14.295979    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:25:14.527210 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:14.527175    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ed7df8-2794-4b08-a6a1-22d6d97eb3ed" path="/var/lib/kubelet/pods/94ed7df8-2794-4b08-a6a1-22d6d97eb3ed/volumes"
Sep 15 21:25:22.011860 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:22.011826    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:22.011860 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:22.011871    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:22.024884 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:22.024852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:22.349425 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:22.349328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:25:49.893886 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.893848    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"]
Sep 15 21:25:49.894313 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.894299    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ed7df8-2794-4b08-a6a1-22d6d97eb3ed" containerName="maas-api"
Sep 15 21:25:49.894313 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.894310    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ed7df8-2794-4b08-a6a1-22d6d97eb3ed" containerName="maas-api"
Sep 15 21:25:49.894418 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.894402    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ed7df8-2794-4b08-a6a1-22d6d97eb3ed" containerName="maas-api"
Sep 15 21:25:49.898043 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.898015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:49.900845 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.900823    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-04c28d\"/\"openshift-service-ca.crt\""
Sep 15 21:25:49.900971 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.900824    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-04c28d\"/\"default-dockercfg-jpt74\""
Sep 15 21:25:49.900971 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.900864    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-04c28d\"/\"kube-root-ca.crt\""
Sep 15 21:25:49.900971 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.900825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-04c28d\"/\"autores-model-04c28d-kserve-self-signed-certs\""
Sep 15 21:25:49.906987 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.906421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"]
Sep 15 21:25:49.968450 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.968414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7x95\" (UniqueName: \"kubernetes.io/projected/c53e4418-2809-4009-a945-f3e083aacc54-kube-api-access-t7x95\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:49.968630 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.968517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-tmp-dir\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:49.968630 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.968565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-dshm\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:49.968780 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.968636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-home\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:49.968893 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.968864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-model-cache\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:49.969040 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:49.968957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53e4418-2809-4009-a945-f3e083aacc54-tls-certs\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.070275 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.070241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-model-cache\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.070275 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.070284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53e4418-2809-4009-a945-f3e083aacc54-tls-certs\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.070529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.070413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7x95\" (UniqueName: \"kubernetes.io/projected/c53e4418-2809-4009-a945-f3e083aacc54-kube-api-access-t7x95\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.070529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.070480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-tmp-dir\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.070529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.070527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-dshm\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.070662 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.070602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-home\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.070720 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.070695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-model-cache\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.070823 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.070806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-tmp-dir\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.070986 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.070966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-home\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.072693 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.072673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-dshm\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.073019 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.073002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53e4418-2809-4009-a945-f3e083aacc54-tls-certs\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.078845 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.078825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7x95\" (UniqueName: \"kubernetes.io/projected/c53e4418-2809-4009-a945-f3e083aacc54-kube-api-access-t7x95\") pod \"autores-model-04c28d-kserve-85664f74c-t7mt7\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") " pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.209310 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.209222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:25:50.490925 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.490895    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"]
Sep 15 21:25:50.492484 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:25:50.492452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc53e4418_2809_4009_a945_f3e083aacc54.slice/crio-ab332d05e3300a9db57156a86035ea9e81a2fd8b47dad774523fc543415dfafa WatchSource:0}: Error finding container ab332d05e3300a9db57156a86035ea9e81a2fd8b47dad774523fc543415dfafa: Status 404 returned error can't find the container with id ab332d05e3300a9db57156a86035ea9e81a2fd8b47dad774523fc543415dfafa
Sep 15 21:25:50.513898 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:50.513871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7" event={"ID":"c53e4418-2809-4009-a945-f3e083aacc54","Type":"ContainerStarted","Data":"ab332d05e3300a9db57156a86035ea9e81a2fd8b47dad774523fc543415dfafa"}
Sep 15 21:25:51.523300 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:51.523262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7" event={"ID":"c53e4418-2809-4009-a945-f3e083aacc54","Type":"ContainerStarted","Data":"e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c"}
Sep 15 21:25:51.542153 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:25:51.542091    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7" podStartSLOduration=2.542074748 podStartE2EDuration="2.542074748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:51.539883152 +0000 UTC m=+941.502940549" watchObservedRunningTime="2026-09-15 21:25:51.542074748 +0000 UTC m=+941.505132188"
Sep 15 21:26:00.210275 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:00.210233    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:26:00.210682 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:00.210299    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:26:00.222427 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:00.222402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:26:00.591781 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:00.591691    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:26:30.136469 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.136433    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"]
Sep 15 21:26:30.141673 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.141648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.146049 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.146030    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cfc1f4\"/\"test-model-cfc1f4-kserve-self-signed-certs\""
Sep 15 21:26:30.146178 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.146047    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cfc1f4\"/\"openshift-service-ca.crt\""
Sep 15 21:26:30.146178 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.146035    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cfc1f4\"/\"default-dockercfg-b6fgl\""
Sep 15 21:26:30.146178 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.146035    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cfc1f4\"/\"kube-root-ca.crt\""
Sep 15 21:26:30.158798 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.158774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"]
Sep 15 21:26:30.299907 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.299873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8gs\" (UniqueName: \"kubernetes.io/projected/1320c907-c1e2-47f3-843b-8627c84e18df-kube-api-access-lm8gs\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.300084 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.299999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1320c907-c1e2-47f3-843b-8627c84e18df-tls-certs\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.300084 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.300034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-model-cache\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.300084 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.300053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-dshm\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.300192 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.300107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-home\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.300192 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.300125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-tmp-dir\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.400877 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.400790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-home\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.400877 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.400830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-tmp-dir\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.401115 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.400881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8gs\" (UniqueName: \"kubernetes.io/projected/1320c907-c1e2-47f3-843b-8627c84e18df-kube-api-access-lm8gs\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.401115 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.401003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1320c907-c1e2-47f3-843b-8627c84e18df-tls-certs\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.401115 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.401030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-model-cache\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.401115 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.401055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-dshm\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.401306 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.401247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-tmp-dir\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.401419 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.401400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-home\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.401555 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.401535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-model-cache\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.403472 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.403451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-dshm\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.403720 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.403702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1320c907-c1e2-47f3-843b-8627c84e18df-tls-certs\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.410183 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.410160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8gs\" (UniqueName: \"kubernetes.io/projected/1320c907-c1e2-47f3-843b-8627c84e18df-kube-api-access-lm8gs\") pod \"test-model-cfc1f4-kserve-5566c4566c-vbqt7\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") " pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.450958 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.450931    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:30.729664 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.729640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"]
Sep 15 21:26:30.731477 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:26:30.731449    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1320c907_c1e2_47f3_843b_8627c84e18df.slice/crio-8bdc9efc82bf84adc6a4ca59d69ce0ec519a2778c3c879dfbee4fea6394ffa88 WatchSource:0}: Error finding container 8bdc9efc82bf84adc6a4ca59d69ce0ec519a2778c3c879dfbee4fea6394ffa88: Status 404 returned error can't find the container with id 8bdc9efc82bf84adc6a4ca59d69ce0ec519a2778c3c879dfbee4fea6394ffa88
Sep 15 21:26:30.774375 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:30.774321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7" event={"ID":"1320c907-c1e2-47f3-843b-8627c84e18df","Type":"ContainerStarted","Data":"8bdc9efc82bf84adc6a4ca59d69ce0ec519a2778c3c879dfbee4fea6394ffa88"}
Sep 15 21:26:31.782274 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:31.782234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7" event={"ID":"1320c907-c1e2-47f3-843b-8627c84e18df","Type":"ContainerStarted","Data":"b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402"}
Sep 15 21:26:31.805530 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:31.805482    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7" podStartSLOduration=1.805467902 podStartE2EDuration="1.805467902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:31.803245624 +0000 UTC m=+981.766303004" watchObservedRunningTime="2026-09-15 21:26:31.805467902 +0000 UTC m=+981.768525283"
Sep 15 21:26:40.451039 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:40.451009    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:40.451594 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:40.451055    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:40.463365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:40.463315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:40.828168 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:40.828081    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"]
Sep 15 21:26:40.828388 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:40.828333    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s" podUID="2e1725cb-3b3e-4613-a67c-d1ec692d1f51" containerName="payload-pre-processing" containerID="cri-o://fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f" gracePeriod=30
Sep 15 21:26:40.863225 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:40.860043    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:26:41.096626 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.096559    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:26:41.209704 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.209676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-plugins-config-volume\") pod \"2e1725cb-3b3e-4613-a67c-d1ec692d1f51\" (UID: \"2e1725cb-3b3e-4613-a67c-d1ec692d1f51\") "
Sep 15 21:26:41.209909 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.209761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2j56\" (UniqueName: \"kubernetes.io/projected/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-kube-api-access-z2j56\") pod \"2e1725cb-3b3e-4613-a67c-d1ec692d1f51\" (UID: \"2e1725cb-3b3e-4613-a67c-d1ec692d1f51\") "
Sep 15 21:26:41.210085 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.210058    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e1725cb-3b3e-4613-a67c-d1ec692d1f51" (UID: "2e1725cb-3b3e-4613-a67c-d1ec692d1f51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:26:41.211962 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.211941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-kube-api-access-z2j56" (OuterVolumeSpecName: "kube-api-access-z2j56") pod "2e1725cb-3b3e-4613-a67c-d1ec692d1f51" (UID: "2e1725cb-3b3e-4613-a67c-d1ec692d1f51"). InnerVolumeSpecName "kube-api-access-z2j56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:26:41.311336 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.311298    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2j56\" (UniqueName: \"kubernetes.io/projected/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-kube-api-access-z2j56\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:26:41.311336 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.311321    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1725cb-3b3e-4613-a67c-d1ec692d1f51-plugins-config-volume\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:26:41.849752 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.849717    2563 generic.go:356] "Generic (PLEG): container finished" podID="2e1725cb-3b3e-4613-a67c-d1ec692d1f51" containerID="fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f" exitCode=0
Sep 15 21:26:41.850234 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.849761    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"
Sep 15 21:26:41.850234 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.849799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s" event={"ID":"2e1725cb-3b3e-4613-a67c-d1ec692d1f51","Type":"ContainerDied","Data":"fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f"}
Sep 15 21:26:41.850234 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.849849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s" event={"ID":"2e1725cb-3b3e-4613-a67c-d1ec692d1f51","Type":"ContainerDied","Data":"2e3f6142a041fcde2888ecb7fc014208d86b35779d63871bf19db03f00da43d9"}
Sep 15 21:26:41.850234 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.849872    2563 scope.go:117] "RemoveContainer" containerID="fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f"
Sep 15 21:26:41.860489 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.860474    2563 scope.go:117] "RemoveContainer" containerID="fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f"
Sep 15 21:26:41.860744 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:26:41.860725    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f\": container with ID starting with fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f not found: ID does not exist" containerID="fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f"
Sep 15 21:26:41.860818 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.860753    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f"} err="failed to get container status \"fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f\": rpc error: code = NotFound desc = could not find container \"fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f\": container with ID starting with fcf5994f6eb948a06164ad654452203fb78a9c7e4d9dcfd1fa96129fc0b2e75f not found: ID does not exist"
Sep 15 21:26:41.873521 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.873494    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"]
Sep 15 21:26:41.877704 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:41.877683    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04c28d-64497897fd-wc62s"]
Sep 15 21:26:42.526816 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:42.526787    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1725cb-3b3e-4613-a67c-d1ec692d1f51" path="/var/lib/kubelet/pods/2e1725cb-3b3e-4613-a67c-d1ec692d1f51/volumes"
Sep 15 21:26:46.437379 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:26:46.437338    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"]
Sep 15 21:27:01.456649 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.456611    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7" podUID="c53e4418-2809-4009-a945-f3e083aacc54" containerName="main" containerID="cri-o://e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c" gracePeriod=45
Sep 15 21:27:01.715629 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.715571    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:27:01.816744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.816714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-model-cache\") pod \"c53e4418-2809-4009-a945-f3e083aacc54\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") "
Sep 15 21:27:01.816744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.816745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-tmp-dir\") pod \"c53e4418-2809-4009-a945-f3e083aacc54\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") "
Sep 15 21:27:01.816943 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.816832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7x95\" (UniqueName: \"kubernetes.io/projected/c53e4418-2809-4009-a945-f3e083aacc54-kube-api-access-t7x95\") pod \"c53e4418-2809-4009-a945-f3e083aacc54\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") "
Sep 15 21:27:01.816943 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.816896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-dshm\") pod \"c53e4418-2809-4009-a945-f3e083aacc54\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") "
Sep 15 21:27:01.816943 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.816923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53e4418-2809-4009-a945-f3e083aacc54-tls-certs\") pod \"c53e4418-2809-4009-a945-f3e083aacc54\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") "
Sep 15 21:27:01.817118 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.816949    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-model-cache" (OuterVolumeSpecName: "model-cache") pod "c53e4418-2809-4009-a945-f3e083aacc54" (UID: "c53e4418-2809-4009-a945-f3e083aacc54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:27:01.817118 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.816968    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-home\") pod \"c53e4418-2809-4009-a945-f3e083aacc54\" (UID: \"c53e4418-2809-4009-a945-f3e083aacc54\") "
Sep 15 21:27:01.817118 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.817075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c53e4418-2809-4009-a945-f3e083aacc54" (UID: "c53e4418-2809-4009-a945-f3e083aacc54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:27:01.817295 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.817261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-home" (OuterVolumeSpecName: "home") pod "c53e4418-2809-4009-a945-f3e083aacc54" (UID: "c53e4418-2809-4009-a945-f3e083aacc54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:27:01.817442 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.817320    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:27:01.817442 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.817333    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:27:01.819181 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.819160    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-dshm" (OuterVolumeSpecName: "dshm") pod "c53e4418-2809-4009-a945-f3e083aacc54" (UID: "c53e4418-2809-4009-a945-f3e083aacc54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:27:01.819270 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.819194    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53e4418-2809-4009-a945-f3e083aacc54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c53e4418-2809-4009-a945-f3e083aacc54" (UID: "c53e4418-2809-4009-a945-f3e083aacc54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:27:01.819270 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.819227    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53e4418-2809-4009-a945-f3e083aacc54-kube-api-access-t7x95" (OuterVolumeSpecName: "kube-api-access-t7x95") pod "c53e4418-2809-4009-a945-f3e083aacc54" (UID: "c53e4418-2809-4009-a945-f3e083aacc54"). InnerVolumeSpecName "kube-api-access-t7x95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:27:01.918556 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.918527    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:27:01.918556 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.918554    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53e4418-2809-4009-a945-f3e083aacc54-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:27:01.918713 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.918564    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53e4418-2809-4009-a945-f3e083aacc54-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:27:01.918713 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.918573    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7x95\" (UniqueName: \"kubernetes.io/projected/c53e4418-2809-4009-a945-f3e083aacc54-kube-api-access-t7x95\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:27:01.971124 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.971060    2563 generic.go:356] "Generic (PLEG): container finished" podID="c53e4418-2809-4009-a945-f3e083aacc54" containerID="e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c" exitCode=0
Sep 15 21:27:01.971124 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.971103    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"
Sep 15 21:27:01.971124 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.971114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7" event={"ID":"c53e4418-2809-4009-a945-f3e083aacc54","Type":"ContainerDied","Data":"e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c"}
Sep 15 21:27:01.971298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.971143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7" event={"ID":"c53e4418-2809-4009-a945-f3e083aacc54","Type":"ContainerDied","Data":"ab332d05e3300a9db57156a86035ea9e81a2fd8b47dad774523fc543415dfafa"}
Sep 15 21:27:01.971298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.971159    2563 scope.go:117] "RemoveContainer" containerID="e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c"
Sep 15 21:27:01.981447 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.981428    2563 scope.go:117] "RemoveContainer" containerID="e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c"
Sep 15 21:27:01.981703 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:27:01.981685    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c\": container with ID starting with e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c not found: ID does not exist" containerID="e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c"
Sep 15 21:27:01.981769 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.981716    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c"} err="failed to get container status \"e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c\": rpc error: code = NotFound desc = could not find container \"e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c\": container with ID starting with e1631e94de7b78051bd7b947f66f0a8f5905da1b5ec23ccd3d77c2752914bd5c not found: ID does not exist"
Sep 15 21:27:01.993101 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.993082    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"]
Sep 15 21:27:01.996910 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:01.996889    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-04c28d/autores-model-04c28d-kserve-85664f74c-t7mt7"]
Sep 15 21:27:02.526930 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:27:02.526893    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c53e4418-2809-4009-a945-f3e083aacc54" path="/var/lib/kubelet/pods/c53e4418-2809-4009-a945-f3e083aacc54/volumes"
Sep 15 21:28:04.508830 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.508754    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"]
Sep 15 21:28:04.509250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.509234    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c53e4418-2809-4009-a945-f3e083aacc54" containerName="main"
Sep 15 21:28:04.509292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.509252    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c53e4418-2809-4009-a945-f3e083aacc54" containerName="main"
Sep 15 21:28:04.509292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.509262    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1725cb-3b3e-4613-a67c-d1ec692d1f51" containerName="payload-pre-processing"
Sep 15 21:28:04.509292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.509267    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1725cb-3b3e-4613-a67c-d1ec692d1f51" containerName="payload-pre-processing"
Sep 15 21:28:04.509408 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.509330    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1725cb-3b3e-4613-a67c-d1ec692d1f51" containerName="payload-pre-processing"
Sep 15 21:28:04.509408 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.509338    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c53e4418-2809-4009-a945-f3e083aacc54" containerName="main"
Sep 15 21:28:04.512701 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.512677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.514654 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.514634    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-318c22\"/\"premium-w0-318c22-kserve-self-signed-certs\""
Sep 15 21:28:04.528385 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.528360    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"]
Sep 15 21:28:04.638838 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.638800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmd9\" (UniqueName: \"kubernetes.io/projected/ae15f1b9-db5b-4f74-b8ee-61cf58147429-kube-api-access-crmd9\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.638838 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.638840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tmp-dir\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.639098 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.638872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tls-certs\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.639098 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.638976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-model-cache\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.639098 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.639040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-dshm\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.639098 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.639084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-home\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.740543 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.740509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-model-cache\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.740718 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.740562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-dshm\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.740718 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.740589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-home\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.740718 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.740627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmd9\" (UniqueName: \"kubernetes.io/projected/ae15f1b9-db5b-4f74-b8ee-61cf58147429-kube-api-access-crmd9\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.740718 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.740651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tmp-dir\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.740718 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.740674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tls-certs\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.740973 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.740948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-model-cache\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.741017 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.740975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tmp-dir\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.741070 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.741048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-home\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.742818 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.742798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-dshm\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.743064 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.743048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tls-certs\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.751061 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.751041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmd9\" (UniqueName: \"kubernetes.io/projected/ae15f1b9-db5b-4f74-b8ee-61cf58147429-kube-api-access-crmd9\") pod \"premium-w0-318c22-kserve-5cd9d7b74c-lpj26\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") " pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:04.823208 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:04.823121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:05.098746 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:05.098716    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"]
Sep 15 21:28:05.100440 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:28:05.100416    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae15f1b9_db5b_4f74_b8ee_61cf58147429.slice/crio-fd3ba984e124bde8e20874373b0ebc93015a3206785e2470735746b02a63c078 WatchSource:0}: Error finding container fd3ba984e124bde8e20874373b0ebc93015a3206785e2470735746b02a63c078: Status 404 returned error can't find the container with id fd3ba984e124bde8e20874373b0ebc93015a3206785e2470735746b02a63c078
Sep 15 21:28:05.350798 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:05.350697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26" event={"ID":"ae15f1b9-db5b-4f74-b8ee-61cf58147429","Type":"ContainerStarted","Data":"29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90"}
Sep 15 21:28:05.350798 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:05.350739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26" event={"ID":"ae15f1b9-db5b-4f74-b8ee-61cf58147429","Type":"ContainerStarted","Data":"fd3ba984e124bde8e20874373b0ebc93015a3206785e2470735746b02a63c078"}
Sep 15 21:28:05.374191 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:05.374140    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26" podStartSLOduration=1.374124811 podStartE2EDuration="1.374124811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:05.37338112 +0000 UTC m=+1075.336438504" watchObservedRunningTime="2026-09-15 21:28:05.374124811 +0000 UTC m=+1075.337182218"
Sep 15 21:28:14.824030 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:14.823978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:14.824552 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:14.824046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:14.836431 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:14.836408    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:15.421956 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:15.421923    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:28:37.809416 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.809381    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"]
Sep 15 21:28:37.813298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.813277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:37.815586 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.815566    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f457e5\"/\"kube-root-ca.crt\""
Sep 15 21:28:37.816340 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.816321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f457e5\"/\"test-model-f457e5-kserve-self-signed-certs\""
Sep 15 21:28:37.816340 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.816326    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f457e5\"/\"openshift-service-ca.crt\""
Sep 15 21:28:37.816518 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.816327    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f457e5\"/\"default-dockercfg-k2prh\""
Sep 15 21:28:37.820560 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.820536    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"]
Sep 15 21:28:37.903739 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.903704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-tmp-dir\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:37.903739 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.903743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svmq2\" (UniqueName: \"kubernetes.io/projected/9c469b68-d775-46f7-b693-466de65efc08-kube-api-access-svmq2\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:37.903957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.903825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c469b68-d775-46f7-b693-466de65efc08-tls-certs\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:37.903957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.903879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-home\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:37.903957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.903909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-model-cache\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:37.903957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:37.903935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-dshm\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.004840 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.004800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-home\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.004840 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.004847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-model-cache\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.005093 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.004867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-dshm\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.005580 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.005003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-tmp-dir\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.006773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.005768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svmq2\" (UniqueName: \"kubernetes.io/projected/9c469b68-d775-46f7-b693-466de65efc08-kube-api-access-svmq2\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.006773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.005866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c469b68-d775-46f7-b693-466de65efc08-tls-certs\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.006773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.005676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-tmp-dir\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.006773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.005633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-home\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.006773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.005658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-model-cache\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.011768 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.009372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-dshm\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.014758 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.014731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svmq2\" (UniqueName: \"kubernetes.io/projected/9c469b68-d775-46f7-b693-466de65efc08-kube-api-access-svmq2\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.015569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.015550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c469b68-d775-46f7-b693-466de65efc08-tls-certs\") pod \"test-model-f457e5-kserve-5b9c7994bf-2wtr6\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") " pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.125000 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.124960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:38.399025 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.398998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"]
Sep 15 21:28:38.400724 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:28:38.400699    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c469b68_d775_46f7_b693_466de65efc08.slice/crio-c714798bc8ed7d7818551084d81412715481f680c57326c55408dac8215c7f55 WatchSource:0}: Error finding container c714798bc8ed7d7818551084d81412715481f680c57326c55408dac8215c7f55: Status 404 returned error can't find the container with id c714798bc8ed7d7818551084d81412715481f680c57326c55408dac8215c7f55
Sep 15 21:28:38.560619 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.560579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6" event={"ID":"9c469b68-d775-46f7-b693-466de65efc08","Type":"ContainerStarted","Data":"d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a"}
Sep 15 21:28:38.560781 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.560624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6" event={"ID":"9c469b68-d775-46f7-b693-466de65efc08","Type":"ContainerStarted","Data":"c714798bc8ed7d7818551084d81412715481f680c57326c55408dac8215c7f55"}
Sep 15 21:28:38.577724 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:38.577672    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6" podStartSLOduration=1.5776570859999999 podStartE2EDuration="1.577657086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:38.576298402 +0000 UTC m=+1108.539355786" watchObservedRunningTime="2026-09-15 21:28:38.577657086 +0000 UTC m=+1108.540714466"
Sep 15 21:28:48.125831 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:48.125790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:48.126304 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:48.125855    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:48.137983 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:48.137961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:28:48.629512 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:28:48.629479    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:30:00.145325 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.145287    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542"]
Sep 15 21:30:00.149977 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.149960    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j"]
Sep 15 21:30:00.150100 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.150080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542"
Sep 15 21:30:00.152438 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.152420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:30:00.153913 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.153893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j"
Sep 15 21:30:00.161727 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.160545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j"]
Sep 15 21:30:00.164147 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.164124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542"]
Sep 15 21:30:00.166224 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.166203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542"
Sep 15 21:30:00.175845 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.175823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j"
Sep 15 21:30:00.453078 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.453037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542"]
Sep 15 21:30:00.454137 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:30:00.454111    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4c4515_f093_4e89_9c5c_2d1559c32f24.slice/crio-d88798d6b9355e0701b0ab0933cbd1b3839cac86670a9dd7fab7390522ff79fd WatchSource:0}: Error finding container d88798d6b9355e0701b0ab0933cbd1b3839cac86670a9dd7fab7390522ff79fd: Status 404 returned error can't find the container with id d88798d6b9355e0701b0ab0933cbd1b3839cac86670a9dd7fab7390522ff79fd
Sep 15 21:30:00.456337 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.456316    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:30:00.457127 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:00.457102    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j"]
Sep 15 21:30:00.457265 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:30:00.457240    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8703060e_0e53_479a_b273_75695a095ad9.slice/crio-9f36e79f7cd5dd21f02fb521dd716d1bcc159f81e5462e0b80f096239164e889 WatchSource:0}: Error finding container 9f36e79f7cd5dd21f02fb521dd716d1bcc159f81e5462e0b80f096239164e889: Status 404 returned error can't find the container with id 9f36e79f7cd5dd21f02fb521dd716d1bcc159f81e5462e0b80f096239164e889
Sep 15 21:30:01.061261 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:01.061217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542" event={"ID":"fc4c4515-f093-4e89-9c5c-2d1559c32f24","Type":"ContainerStarted","Data":"d88798d6b9355e0701b0ab0933cbd1b3839cac86670a9dd7fab7390522ff79fd"}
Sep 15 21:30:01.062703 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:01.062672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j" event={"ID":"8703060e-0e53-479a-b273-75695a095ad9","Type":"ContainerStarted","Data":"9f36e79f7cd5dd21f02fb521dd716d1bcc159f81e5462e0b80f096239164e889"}
Sep 15 21:30:03.077095 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:03.076998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542" event={"ID":"fc4c4515-f093-4e89-9c5c-2d1559c32f24","Type":"ContainerStarted","Data":"749cb283191cfb431c717b3fc3b9ecfee4f03e1ac071de553dc679a30cac8c0b"}
Sep 15 21:30:03.078683 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:03.078658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j" event={"ID":"8703060e-0e53-479a-b273-75695a095ad9","Type":"ContainerStarted","Data":"d1d9128d0aebda1a6f81e977d834b6631104474ea61c2ced2197d90f1744481b"}
Sep 15 21:30:03.092293 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:03.092247    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542" podStartSLOduration=1.040115061 podStartE2EDuration="3.092233682s" totalImagesPullingTime="2.052118621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:03.090048365 +0000 UTC m=+1193.053105746" watchObservedRunningTime="2026-09-15 21:30:03.092233682 +0000 UTC m=+1193.055291106"
Sep 15 21:30:03.105550 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:03.105506    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j" podStartSLOduration=1.051897719 podStartE2EDuration="3.105491704s" totalImagesPullingTime="2.053593985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:03.103406987 +0000 UTC m=+1193.066464370" watchObservedRunningTime="2026-09-15 21:30:03.105491704 +0000 UTC m=+1193.068549084"
Sep 15 21:30:04.086594 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:04.086557    2563 generic.go:356] "Generic (PLEG): container finished" podID="fc4c4515-f093-4e89-9c5c-2d1559c32f24" containerID="749cb283191cfb431c717b3fc3b9ecfee4f03e1ac071de553dc679a30cac8c0b" exitCode=0
Sep 15 21:30:04.087014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:04.086643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542" event={"ID":"fc4c4515-f093-4e89-9c5c-2d1559c32f24","Type":"ContainerDied","Data":"749cb283191cfb431c717b3fc3b9ecfee4f03e1ac071de553dc679a30cac8c0b"}
Sep 15 21:30:04.088133 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:04.088110    2563 generic.go:356] "Generic (PLEG): container finished" podID="8703060e-0e53-479a-b273-75695a095ad9" containerID="d1d9128d0aebda1a6f81e977d834b6631104474ea61c2ced2197d90f1744481b" exitCode=0
Sep 15 21:30:04.088249 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:04.088167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j" event={"ID":"8703060e-0e53-479a-b273-75695a095ad9","Type":"ContainerDied","Data":"d1d9128d0aebda1a6f81e977d834b6631104474ea61c2ced2197d90f1744481b"}
Sep 15 21:30:05.256853 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:05.256820    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j"
Sep 15 21:30:05.259730 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:05.259706    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542"
Sep 15 21:30:06.103445 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:06.103412    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542"
Sep 15 21:30:06.103638 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:06.103442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542" event={"ID":"fc4c4515-f093-4e89-9c5c-2d1559c32f24","Type":"ContainerDied","Data":"d88798d6b9355e0701b0ab0933cbd1b3839cac86670a9dd7fab7390522ff79fd"}
Sep 15 21:30:06.103638 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:06.103504    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d88798d6b9355e0701b0ab0933cbd1b3839cac86670a9dd7fab7390522ff79fd"
Sep 15 21:30:06.104972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:06.104950    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j"
Sep 15 21:30:06.105092 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:06.104946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-9vz4j" event={"ID":"8703060e-0e53-479a-b273-75695a095ad9","Type":"ContainerDied","Data":"9f36e79f7cd5dd21f02fb521dd716d1bcc159f81e5462e0b80f096239164e889"}
Sep 15 21:30:06.105154 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:06.105098    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f36e79f7cd5dd21f02fb521dd716d1bcc159f81e5462e0b80f096239164e889"
Sep 15 21:30:15.291408 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.291370    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"]
Sep 15 21:30:15.291843 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.291821    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8703060e-0e53-479a-b273-75695a095ad9" containerName="cleanup"
Sep 15 21:30:15.291843 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.291834    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8703060e-0e53-479a-b273-75695a095ad9" containerName="cleanup"
Sep 15 21:30:15.291920 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.291858    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4c4515-f093-4e89-9c5c-2d1559c32f24" containerName="cleanup"
Sep 15 21:30:15.291920 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.291865    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4c4515-f093-4e89-9c5c-2d1559c32f24" containerName="cleanup"
Sep 15 21:30:15.291986 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.291935    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8703060e-0e53-479a-b273-75695a095ad9" containerName="cleanup"
Sep 15 21:30:15.291986 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.291946    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4c4515-f093-4e89-9c5c-2d1559c32f24" containerName="cleanup"
Sep 15 21:30:15.295376 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.295315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.298741 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.298720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f0d7d5\"/\"noten-model-f0d7d5-kserve-self-signed-certs\""
Sep 15 21:30:15.298866 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.298728    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f0d7d5\"/\"default-dockercfg-4rzkc\""
Sep 15 21:30:15.298866 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.298766    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f0d7d5\"/\"kube-root-ca.crt\""
Sep 15 21:30:15.298866 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.298772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f0d7d5\"/\"openshift-service-ca.crt\""
Sep 15 21:30:15.303999 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.303976    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"]
Sep 15 21:30:15.319433 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.319405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-dshm\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.319588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.319444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808d4557-d2dd-40d1-be92-d605914669c8-tls-certs\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.319588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.319485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-model-cache\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.319588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.319514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wksrb\" (UniqueName: \"kubernetes.io/projected/808d4557-d2dd-40d1-be92-d605914669c8-kube-api-access-wksrb\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.319588 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.319538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-tmp-dir\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.319735 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.319646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-home\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.420623 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.420588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-home\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.420806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.420711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-dshm\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.420806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.420745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808d4557-d2dd-40d1-be92-d605914669c8-tls-certs\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.420806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.420782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-model-cache\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.420975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.420844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wksrb\" (UniqueName: \"kubernetes.io/projected/808d4557-d2dd-40d1-be92-d605914669c8-kube-api-access-wksrb\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.420975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.420894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-tmp-dir\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.421076 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.421064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-model-cache\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.421133 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.421064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-home\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.421287 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.421267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-tmp-dir\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.423142 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.423115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-dshm\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.423447 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.423427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808d4557-d2dd-40d1-be92-d605914669c8-tls-certs\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.429415 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.429394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wksrb\" (UniqueName: \"kubernetes.io/projected/808d4557-d2dd-40d1-be92-d605914669c8-kube-api-access-wksrb\") pod \"noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") " pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.608110 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.607761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:15.909578 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:30:15.909536    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod808d4557_d2dd_40d1_be92_d605914669c8.slice/crio-a195a39a022c1c4fa4ffdcbbdc4bc19b9331233745e9055ddedfc67326993c6f WatchSource:0}: Error finding container a195a39a022c1c4fa4ffdcbbdc4bc19b9331233745e9055ddedfc67326993c6f: Status 404 returned error can't find the container with id a195a39a022c1c4fa4ffdcbbdc4bc19b9331233745e9055ddedfc67326993c6f
Sep 15 21:30:15.910701 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:15.910661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"]
Sep 15 21:30:16.172093 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:16.172000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5" event={"ID":"808d4557-d2dd-40d1-be92-d605914669c8","Type":"ContainerStarted","Data":"bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54"}
Sep 15 21:30:16.172093 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:16.172049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5" event={"ID":"808d4557-d2dd-40d1-be92-d605914669c8","Type":"ContainerStarted","Data":"a195a39a022c1c4fa4ffdcbbdc4bc19b9331233745e9055ddedfc67326993c6f"}
Sep 15 21:30:16.188310 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:16.188248    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5" podStartSLOduration=1.188227486 podStartE2EDuration="1.188227486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:16.187996153 +0000 UTC m=+1206.151053534" watchObservedRunningTime="2026-09-15 21:30:16.188227486 +0000 UTC m=+1206.151284869"
Sep 15 21:30:25.608623 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:25.608589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:25.608623 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:25.608638    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:25.621234 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:25.621209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:26.263013 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:26.262984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:30:36.778378 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:36.778332    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7851-b5b7bb"]
Sep 15 21:30:36.784469 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:36.784437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-b5b7bb"
Sep 15 21:30:36.786544 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:36.786505    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncvz5\""
Sep 15 21:30:36.788470 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:36.788446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-b5b7bb"]
Sep 15 21:30:36.956453 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:36.956413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scw7\" (UniqueName: \"kubernetes.io/projected/681471f0-607c-4872-9305-399f2dfd87ae-kube-api-access-5scw7\") pod \"test-curl-7851-b5b7bb\" (UID: \"681471f0-607c-4872-9305-399f2dfd87ae\") " pod="opendatahub/test-curl-7851-b5b7bb"
Sep 15 21:30:37.057713 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:37.057625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scw7\" (UniqueName: \"kubernetes.io/projected/681471f0-607c-4872-9305-399f2dfd87ae-kube-api-access-5scw7\") pod \"test-curl-7851-b5b7bb\" (UID: \"681471f0-607c-4872-9305-399f2dfd87ae\") " pod="opendatahub/test-curl-7851-b5b7bb"
Sep 15 21:30:37.066136 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:37.066103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scw7\" (UniqueName: \"kubernetes.io/projected/681471f0-607c-4872-9305-399f2dfd87ae-kube-api-access-5scw7\") pod \"test-curl-7851-b5b7bb\" (UID: \"681471f0-607c-4872-9305-399f2dfd87ae\") " pod="opendatahub/test-curl-7851-b5b7bb"
Sep 15 21:30:37.097722 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:37.097698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-b5b7bb"
Sep 15 21:30:37.370773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:37.370747    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-b5b7bb"]
Sep 15 21:30:37.371376 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:30:37.371328    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod681471f0_607c_4872_9305_399f2dfd87ae.slice/crio-5909304c2d61e934756b4992d11f599d3b8358a14a9d88a0976d67259925e2bb WatchSource:0}: Error finding container 5909304c2d61e934756b4992d11f599d3b8358a14a9d88a0976d67259925e2bb: Status 404 returned error can't find the container with id 5909304c2d61e934756b4992d11f599d3b8358a14a9d88a0976d67259925e2bb
Sep 15 21:30:38.327654 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:38.327619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-b5b7bb" event={"ID":"681471f0-607c-4872-9305-399f2dfd87ae","Type":"ContainerStarted","Data":"3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c"}
Sep 15 21:30:38.328013 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:38.327670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-b5b7bb" event={"ID":"681471f0-607c-4872-9305-399f2dfd87ae","Type":"ContainerStarted","Data":"5909304c2d61e934756b4992d11f599d3b8358a14a9d88a0976d67259925e2bb"}
Sep 15 21:30:38.342750 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:38.342703    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7851-b5b7bb" podStartSLOduration=2.342687388 podStartE2EDuration="2.342687388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:38.341714645 +0000 UTC m=+1228.304772028" watchObservedRunningTime="2026-09-15 21:30:38.342687388 +0000 UTC m=+1228.305744768"
Sep 15 21:30:38.931918 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:38.931883    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7851-b5b7bb"]
Sep 15 21:30:38.936580 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:38.936538    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7851-b5b7bb"]
Sep 15 21:30:39.059238 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.059157    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7851-adf856"]
Sep 15 21:30:39.059644 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.059629    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="681471f0-607c-4872-9305-399f2dfd87ae" containerName="test-curl-7851-b5b7bb"
Sep 15 21:30:39.059699 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.059645    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="681471f0-607c-4872-9305-399f2dfd87ae" containerName="test-curl-7851-b5b7bb"
Sep 15 21:30:39.059734 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.059724    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="681471f0-607c-4872-9305-399f2dfd87ae" containerName="test-curl-7851-b5b7bb"
Sep 15 21:30:39.061888 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.061867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:39.069226 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.069199    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-adf856"]
Sep 15 21:30:39.179221 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.179185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trjcs\" (UniqueName: \"kubernetes.io/projected/702eb8fe-af3d-46f4-8016-67062580c2a3-kube-api-access-trjcs\") pod \"test-curl-7851-adf856\" (UID: \"702eb8fe-af3d-46f4-8016-67062580c2a3\") " pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:39.280406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.280332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trjcs\" (UniqueName: \"kubernetes.io/projected/702eb8fe-af3d-46f4-8016-67062580c2a3-kube-api-access-trjcs\") pod \"test-curl-7851-adf856\" (UID: \"702eb8fe-af3d-46f4-8016-67062580c2a3\") " pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:39.289560 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.289542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trjcs\" (UniqueName: \"kubernetes.io/projected/702eb8fe-af3d-46f4-8016-67062580c2a3-kube-api-access-trjcs\") pod \"test-curl-7851-adf856\" (UID: \"702eb8fe-af3d-46f4-8016-67062580c2a3\") " pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:39.335307 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:39.335221    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7851-b5b7bb" container="test-curl-7851-b5b7bb"
Sep 15 21:30:39.373778 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.373751    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:39.643704 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:39.643676    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-adf856"]
Sep 15 21:30:39.644637 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:30:39.644616    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod702eb8fe_af3d_46f4_8016_67062580c2a3.slice/crio-cd0257e5d45a29fc10041bb3963a7ebce2377e807964274ca5011120434dfb2c WatchSource:0}: Error finding container cd0257e5d45a29fc10041bb3963a7ebce2377e807964274ca5011120434dfb2c: Status 404 returned error can't find the container with id cd0257e5d45a29fc10041bb3963a7ebce2377e807964274ca5011120434dfb2c
Sep 15 21:30:40.342972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:40.342935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-adf856" event={"ID":"702eb8fe-af3d-46f4-8016-67062580c2a3","Type":"ContainerStarted","Data":"b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962"}
Sep 15 21:30:40.343413 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:40.342981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-adf856" event={"ID":"702eb8fe-af3d-46f4-8016-67062580c2a3","Type":"ContainerStarted","Data":"cd0257e5d45a29fc10041bb3963a7ebce2377e807964274ca5011120434dfb2c"}
Sep 15 21:30:40.343413 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:40.343084    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7851-b5b7bb" podUID="681471f0-607c-4872-9305-399f2dfd87ae" containerName="test-curl-7851-b5b7bb" containerID="cri-o://3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c" gracePeriod=2
Sep 15 21:30:40.345542 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:40.345520    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-b5b7bb\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="681471f0-607c-4872-9305-399f2dfd87ae" pod="opendatahub/test-curl-7851-b5b7bb"
Sep 15 21:30:40.358748 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:40.358707    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7851-adf856" podStartSLOduration=1.358691667 podStartE2EDuration="1.358691667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:40.35820521 +0000 UTC m=+1230.321262593" watchObservedRunningTime="2026-09-15 21:30:40.358691667 +0000 UTC m=+1230.321749050"
Sep 15 21:30:40.359946 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:40.359921    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-b5b7bb\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="681471f0-607c-4872-9305-399f2dfd87ae" pod="opendatahub/test-curl-7851-b5b7bb"
Sep 15 21:30:40.525599 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:40.525565    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-b5b7bb\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="681471f0-607c-4872-9305-399f2dfd87ae" pod="opendatahub/test-curl-7851-b5b7bb"
Sep 15 21:30:40.939087 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:40.939031    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7851-adf856"]
Sep 15 21:30:40.943765 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:40.943736    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7851-adf856"]
Sep 15 21:30:41.061316 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.061234    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7851-f62586"]
Sep 15 21:30:41.061701 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.061688    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="702eb8fe-af3d-46f4-8016-67062580c2a3" containerName="test-curl-7851-adf856"
Sep 15 21:30:41.061749 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.061703    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="702eb8fe-af3d-46f4-8016-67062580c2a3" containerName="test-curl-7851-adf856"
Sep 15 21:30:41.061815 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.061806    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="702eb8fe-af3d-46f4-8016-67062580c2a3" containerName="test-curl-7851-adf856"
Sep 15 21:30:41.063983 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.063966    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-f62586"
Sep 15 21:30:41.071896 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.071874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-f62586"]
Sep 15 21:30:41.202255 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.202218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6ts\" (UniqueName: \"kubernetes.io/projected/bc22d1a5-55d5-455a-8b56-8f98c732b9a4-kube-api-access-kp6ts\") pod \"test-curl-7851-f62586\" (UID: \"bc22d1a5-55d5-455a-8b56-8f98c732b9a4\") " pod="opendatahub/test-curl-7851-f62586"
Sep 15 21:30:41.310842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.304150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6ts\" (UniqueName: \"kubernetes.io/projected/bc22d1a5-55d5-455a-8b56-8f98c732b9a4-kube-api-access-kp6ts\") pod \"test-curl-7851-f62586\" (UID: \"bc22d1a5-55d5-455a-8b56-8f98c732b9a4\") " pod="opendatahub/test-curl-7851-f62586"
Sep 15 21:30:41.313174 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.313099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6ts\" (UniqueName: \"kubernetes.io/projected/bc22d1a5-55d5-455a-8b56-8f98c732b9a4-kube-api-access-kp6ts\") pod \"test-curl-7851-f62586\" (UID: \"bc22d1a5-55d5-455a-8b56-8f98c732b9a4\") " pod="opendatahub/test-curl-7851-f62586"
Sep 15 21:30:41.349422 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:41.349396    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7851-adf856" container="test-curl-7851-adf856"
Sep 15 21:30:41.378369 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.378327    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-f62586"
Sep 15 21:30:41.652059 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:41.652027    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-f62586"]
Sep 15 21:30:41.657854 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:30:41.657828    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc22d1a5_55d5_455a_8b56_8f98c732b9a4.slice/crio-539447c010d286803df455fdb156c37eb0dbdeb9fd21f04e4c5c9958e98906f8 WatchSource:0}: Error finding container 539447c010d286803df455fdb156c37eb0dbdeb9fd21f04e4c5c9958e98906f8: Status 404 returned error can't find the container with id 539447c010d286803df455fdb156c37eb0dbdeb9fd21f04e4c5c9958e98906f8
Sep 15 21:30:42.357142 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:42.357103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-f62586" event={"ID":"bc22d1a5-55d5-455a-8b56-8f98c732b9a4","Type":"ContainerStarted","Data":"b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead"}
Sep 15 21:30:42.357561 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:42.357153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-f62586" event={"ID":"bc22d1a5-55d5-455a-8b56-8f98c732b9a4","Type":"ContainerStarted","Data":"539447c010d286803df455fdb156c37eb0dbdeb9fd21f04e4c5c9958e98906f8"}
Sep 15 21:30:42.357561 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:42.357235    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7851-adf856" podUID="702eb8fe-af3d-46f4-8016-67062580c2a3" containerName="test-curl-7851-adf856" containerID="cri-o://b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962" gracePeriod=2
Sep 15 21:30:42.381371 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:42.381318    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-adf856\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="702eb8fe-af3d-46f4-8016-67062580c2a3" pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:42.381557 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:42.381518    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7851-f62586" podStartSLOduration=1.381503495 podStartE2EDuration="1.381503495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:42.379105803 +0000 UTC m=+1232.342163184" watchObservedRunningTime="2026-09-15 21:30:42.381503495 +0000 UTC m=+1232.344560877"
Sep 15 21:30:42.383310 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:42.383285    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-adf856\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="702eb8fe-af3d-46f4-8016-67062580c2a3" pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:42.512387 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:42.512341    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-b5b7bb"
Sep 15 21:30:42.514069 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:42.514045    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-adf856\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="702eb8fe-af3d-46f4-8016-67062580c2a3" pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:42.617691 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:42.617623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5scw7\" (UniqueName: \"kubernetes.io/projected/681471f0-607c-4872-9305-399f2dfd87ae-kube-api-access-5scw7\") pod \"681471f0-607c-4872-9305-399f2dfd87ae\" (UID: \"681471f0-607c-4872-9305-399f2dfd87ae\") "
Sep 15 21:30:42.619495 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:42.619472    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681471f0-607c-4872-9305-399f2dfd87ae-kube-api-access-5scw7" (OuterVolumeSpecName: "kube-api-access-5scw7") pod "681471f0-607c-4872-9305-399f2dfd87ae" (UID: "681471f0-607c-4872-9305-399f2dfd87ae"). InnerVolumeSpecName "kube-api-access-5scw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:42.719026 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:42.719000    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5scw7\" (UniqueName: \"kubernetes.io/projected/681471f0-607c-4872-9305-399f2dfd87ae-kube-api-access-5scw7\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:30:42.973232 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:42.973191    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7851-f62586"]
Sep 15 21:30:42.977608 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:42.977557    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7851-f62586"]
Sep 15 21:30:43.094583 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.094549    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7851-50dcaf"]
Sep 15 21:30:43.095013 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.095000    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc22d1a5-55d5-455a-8b56-8f98c732b9a4" containerName="test-curl-7851-f62586"
Sep 15 21:30:43.095056 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.095015    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc22d1a5-55d5-455a-8b56-8f98c732b9a4" containerName="test-curl-7851-f62586"
Sep 15 21:30:43.095112 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.095104    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc22d1a5-55d5-455a-8b56-8f98c732b9a4" containerName="test-curl-7851-f62586"
Sep 15 21:30:43.097352 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.097328    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-50dcaf"
Sep 15 21:30:43.102428 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.102366    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-50dcaf"]
Sep 15 21:30:43.122544 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.122520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rrw\" (UniqueName: \"kubernetes.io/projected/7436e203-13f4-42e0-8f07-646371949a00-kube-api-access-s8rrw\") pod \"test-curl-7851-50dcaf\" (UID: \"7436e203-13f4-42e0-8f07-646371949a00\") " pod="opendatahub/test-curl-7851-50dcaf"
Sep 15 21:30:43.223483 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.223403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rrw\" (UniqueName: \"kubernetes.io/projected/7436e203-13f4-42e0-8f07-646371949a00-kube-api-access-s8rrw\") pod \"test-curl-7851-50dcaf\" (UID: \"7436e203-13f4-42e0-8f07-646371949a00\") " pod="opendatahub/test-curl-7851-50dcaf"
Sep 15 21:30:43.231727 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.231705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rrw\" (UniqueName: \"kubernetes.io/projected/7436e203-13f4-42e0-8f07-646371949a00-kube-api-access-s8rrw\") pod \"test-curl-7851-50dcaf\" (UID: \"7436e203-13f4-42e0-8f07-646371949a00\") " pod="opendatahub/test-curl-7851-50dcaf"
Sep 15 21:30:43.364870 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.364835    2563 generic.go:356] "Generic (PLEG): container finished" podID="681471f0-607c-4872-9305-399f2dfd87ae" containerID="3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c" exitCode=137
Sep 15 21:30:43.365299 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.364878    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-b5b7bb"
Sep 15 21:30:43.365299 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.364928    2563 scope.go:117] "RemoveContainer" containerID="3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c"
Sep 15 21:30:43.365299 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:43.365164    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7851-f62586" container="test-curl-7851-f62586"
Sep 15 21:30:43.374428 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.374414    2563 scope.go:117] "RemoveContainer" containerID="3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c"
Sep 15 21:30:43.374694 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:43.374676    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c\": container with ID starting with 3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c not found: ID does not exist" containerID="3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c"
Sep 15 21:30:43.374747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.374701    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c"} err="failed to get container status \"3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c\": rpc error: code = NotFound desc = could not find container \"3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c\": container with ID starting with 3b54491d4f21ad4292696ae4f408bc8b2ae8e77e0374008318ca1db039e19e7c not found: ID does not exist"
Sep 15 21:30:43.408026 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.408002    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-50dcaf"
Sep 15 21:30:43.680967 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:43.680939    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-50dcaf"]
Sep 15 21:30:43.681725 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:30:43.681696    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7436e203_13f4_42e0_8f07_646371949a00.slice/crio-aa56a6d1ead8412a43fa4ad1c3f83998a79cea7088f9bc202f8bbd059caac72a WatchSource:0}: Error finding container aa56a6d1ead8412a43fa4ad1c3f83998a79cea7088f9bc202f8bbd059caac72a: Status 404 returned error can't find the container with id aa56a6d1ead8412a43fa4ad1c3f83998a79cea7088f9bc202f8bbd059caac72a
Sep 15 21:30:44.374936 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:44.374907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-50dcaf" event={"ID":"7436e203-13f4-42e0-8f07-646371949a00","Type":"ContainerStarted","Data":"33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16"}
Sep 15 21:30:44.375257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:44.374951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-50dcaf" event={"ID":"7436e203-13f4-42e0-8f07-646371949a00","Type":"ContainerStarted","Data":"aa56a6d1ead8412a43fa4ad1c3f83998a79cea7088f9bc202f8bbd059caac72a"}
Sep 15 21:30:44.375257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:44.375070    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7851-f62586" podUID="bc22d1a5-55d5-455a-8b56-8f98c732b9a4" containerName="test-curl-7851-f62586" containerID="cri-o://b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead" gracePeriod=2
Sep 15 21:30:44.390253 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:44.390226    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-f62586\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="bc22d1a5-55d5-455a-8b56-8f98c732b9a4" pod="opendatahub/test-curl-7851-f62586"
Sep 15 21:30:44.391586 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:44.391546    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7851-50dcaf" podStartSLOduration=1.391531037 podStartE2EDuration="1.391531037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:44.388591437 +0000 UTC m=+1234.351648819" watchObservedRunningTime="2026-09-15 21:30:44.391531037 +0000 UTC m=+1234.354588418"
Sep 15 21:30:44.530592 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:44.530559    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="681471f0-607c-4872-9305-399f2dfd87ae" path="/var/lib/kubelet/pods/681471f0-607c-4872-9305-399f2dfd87ae/volumes"
Sep 15 21:30:44.539020 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:44.538998    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:44.639952 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:44.639926    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trjcs\" (UniqueName: \"kubernetes.io/projected/702eb8fe-af3d-46f4-8016-67062580c2a3-kube-api-access-trjcs\") pod \"702eb8fe-af3d-46f4-8016-67062580c2a3\" (UID: \"702eb8fe-af3d-46f4-8016-67062580c2a3\") "
Sep 15 21:30:44.641732 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:44.641710    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702eb8fe-af3d-46f4-8016-67062580c2a3-kube-api-access-trjcs" (OuterVolumeSpecName: "kube-api-access-trjcs") pod "702eb8fe-af3d-46f4-8016-67062580c2a3" (UID: "702eb8fe-af3d-46f4-8016-67062580c2a3"). InnerVolumeSpecName "kube-api-access-trjcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:44.740846 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:44.740816    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trjcs\" (UniqueName: \"kubernetes.io/projected/702eb8fe-af3d-46f4-8016-67062580c2a3-kube-api-access-trjcs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:30:44.930915 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:44.930839    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"]
Sep 15 21:30:45.003225 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:45.003189    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7851-50dcaf"]
Sep 15 21:30:45.009129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:45.009102    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7851-50dcaf"]
Sep 15 21:30:45.382587 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:45.382552    2563 generic.go:356] "Generic (PLEG): container finished" podID="702eb8fe-af3d-46f4-8016-67062580c2a3" containerID="b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962" exitCode=137
Sep 15 21:30:45.382998 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:45.382607    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:45.382998 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:45.382649    2563 scope.go:117] "RemoveContainer" containerID="b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962"
Sep 15 21:30:45.382998 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:45.382881    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7851-50dcaf" container="test-curl-7851-50dcaf"
Sep 15 21:30:45.392258 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:45.392237    2563 scope.go:117] "RemoveContainer" containerID="b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962"
Sep 15 21:30:45.392549 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:45.392531    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962\": container with ID starting with b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962 not found: ID does not exist" containerID="b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962"
Sep 15 21:30:45.392606 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:45.392557    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962"} err="failed to get container status \"b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962\": rpc error: code = NotFound desc = could not find container \"b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962\": container with ID starting with b8f0f923791aca4d7f3963aa7c4b9dabb14c1cc571a71fca58582f1d0fa49962 not found: ID does not exist"
Sep 15 21:30:45.392867 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:45.392845    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-adf856\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="702eb8fe-af3d-46f4-8016-67062580c2a3" pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:46.390724 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:46.390638    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7851-50dcaf" podUID="7436e203-13f4-42e0-8f07-646371949a00" containerName="test-curl-7851-50dcaf" containerID="cri-o://33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16" gracePeriod=2
Sep 15 21:30:46.392704 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:46.392668    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-adf856\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="702eb8fe-af3d-46f4-8016-67062580c2a3" pod="opendatahub/test-curl-7851-adf856"
Sep 15 21:30:46.394250 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:46.394224    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-50dcaf\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="7436e203-13f4-42e0-8f07-646371949a00" pod="opendatahub/test-curl-7851-50dcaf"
Sep 15 21:30:46.526323 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:46.526287    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="702eb8fe-af3d-46f4-8016-67062580c2a3" path="/var/lib/kubelet/pods/702eb8fe-af3d-46f4-8016-67062580c2a3/volumes"
Sep 15 21:30:46.562003 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:46.561983    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-f62586"
Sep 15 21:30:46.659044 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:46.658960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp6ts\" (UniqueName: \"kubernetes.io/projected/bc22d1a5-55d5-455a-8b56-8f98c732b9a4-kube-api-access-kp6ts\") pod \"bc22d1a5-55d5-455a-8b56-8f98c732b9a4\" (UID: \"bc22d1a5-55d5-455a-8b56-8f98c732b9a4\") "
Sep 15 21:30:46.660995 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:46.660965    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc22d1a5-55d5-455a-8b56-8f98c732b9a4-kube-api-access-kp6ts" (OuterVolumeSpecName: "kube-api-access-kp6ts") pod "bc22d1a5-55d5-455a-8b56-8f98c732b9a4" (UID: "bc22d1a5-55d5-455a-8b56-8f98c732b9a4"). InnerVolumeSpecName "kube-api-access-kp6ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:46.760325 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:46.760299    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp6ts\" (UniqueName: \"kubernetes.io/projected/bc22d1a5-55d5-455a-8b56-8f98c732b9a4-kube-api-access-kp6ts\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:30:47.397941 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:47.397907    2563 generic.go:356] "Generic (PLEG): container finished" podID="bc22d1a5-55d5-455a-8b56-8f98c732b9a4" containerID="b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead" exitCode=137
Sep 15 21:30:47.398322 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:47.397950    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-f62586"
Sep 15 21:30:47.398322 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:47.398004    2563 scope.go:117] "RemoveContainer" containerID="b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead"
Sep 15 21:30:47.407378 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:47.407361    2563 scope.go:117] "RemoveContainer" containerID="b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead"
Sep 15 21:30:47.407623 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:47.407595    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-f62586\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="bc22d1a5-55d5-455a-8b56-8f98c732b9a4" pod="opendatahub/test-curl-7851-f62586"
Sep 15 21:30:47.407720 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:47.407643    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead\": container with ID starting with b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead not found: ID does not exist" containerID="b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead"
Sep 15 21:30:47.407720 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:47.407665    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead"} err="failed to get container status \"b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead\": rpc error: code = NotFound desc = could not find container \"b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead\": container with ID starting with b95787280db6d7bc53eb98c1053c29561590a65633bde9666a0c84bb50862ead not found: ID does not exist"
Sep 15 21:30:48.529218 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:48.529180    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc22d1a5-55d5-455a-8b56-8f98c732b9a4" path="/var/lib/kubelet/pods/bc22d1a5-55d5-455a-8b56-8f98c732b9a4/volumes"
Sep 15 21:30:48.562363 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:48.562329    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-50dcaf"
Sep 15 21:30:48.577648 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:48.577629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8rrw\" (UniqueName: \"kubernetes.io/projected/7436e203-13f4-42e0-8f07-646371949a00-kube-api-access-s8rrw\") pod \"7436e203-13f4-42e0-8f07-646371949a00\" (UID: \"7436e203-13f4-42e0-8f07-646371949a00\") "
Sep 15 21:30:48.579633 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:48.579609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7436e203-13f4-42e0-8f07-646371949a00-kube-api-access-s8rrw" (OuterVolumeSpecName: "kube-api-access-s8rrw") pod "7436e203-13f4-42e0-8f07-646371949a00" (UID: "7436e203-13f4-42e0-8f07-646371949a00"). InnerVolumeSpecName "kube-api-access-s8rrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:48.679488 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:48.679405    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8rrw\" (UniqueName: \"kubernetes.io/projected/7436e203-13f4-42e0-8f07-646371949a00-kube-api-access-s8rrw\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:30:49.413193 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:49.413163    2563 generic.go:356] "Generic (PLEG): container finished" podID="7436e203-13f4-42e0-8f07-646371949a00" containerID="33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16" exitCode=137
Sep 15 21:30:49.413434 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:49.413205    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-50dcaf"
Sep 15 21:30:49.413434 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:49.413290    2563 scope.go:117] "RemoveContainer" containerID="33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16"
Sep 15 21:30:49.423326 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:49.423310    2563 scope.go:117] "RemoveContainer" containerID="33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16"
Sep 15 21:30:49.423440 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:49.423413    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-50dcaf\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="7436e203-13f4-42e0-8f07-646371949a00" pod="opendatahub/test-curl-7851-50dcaf"
Sep 15 21:30:49.423629 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:49.423609    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16\": container with ID starting with 33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16 not found: ID does not exist" containerID="33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16"
Sep 15 21:30:49.423681 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:49.423636    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16"} err="failed to get container status \"33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16\": rpc error: code = NotFound desc = could not find container \"33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16\": container with ID starting with 33e4f32f769bddef5043455c3dfd26c81c3a506db60813c56823314a12f08a16 not found: ID does not exist"
Sep 15 21:30:50.524519 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:30:50.524481    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-50dcaf\" is forbidden: User \"system:node:ip-10-0-143-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-191.ec2.internal' and this object" podUID="7436e203-13f4-42e0-8f07-646371949a00" pod="opendatahub/test-curl-7851-50dcaf"
Sep 15 21:30:50.526302 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:50.526271    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7436e203-13f4-42e0-8f07-646371949a00" path="/var/lib/kubelet/pods/7436e203-13f4-42e0-8f07-646371949a00/volumes"
Sep 15 21:30:57.505908 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:57.505878    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"]
Sep 15 21:30:59.953956 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:30:59.953899    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7" podUID="1320c907-c1e2-47f3-843b-8627c84e18df" containerName="main" containerID="cri-o://b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402" gracePeriod=45
Sep 15 21:31:00.211175 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.211120    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:31:00.299366 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.299304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm8gs\" (UniqueName: \"kubernetes.io/projected/1320c907-c1e2-47f3-843b-8627c84e18df-kube-api-access-lm8gs\") pod \"1320c907-c1e2-47f3-843b-8627c84e18df\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") "
Sep 15 21:31:00.299524 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.299386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-dshm\") pod \"1320c907-c1e2-47f3-843b-8627c84e18df\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") "
Sep 15 21:31:00.299524 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.299421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-model-cache\") pod \"1320c907-c1e2-47f3-843b-8627c84e18df\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") "
Sep 15 21:31:00.299524 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.299518    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-home\") pod \"1320c907-c1e2-47f3-843b-8627c84e18df\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") "
Sep 15 21:31:00.299648 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.299579    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-tmp-dir\") pod \"1320c907-c1e2-47f3-843b-8627c84e18df\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") "
Sep 15 21:31:00.299648 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.299624    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1320c907-c1e2-47f3-843b-8627c84e18df-tls-certs\") pod \"1320c907-c1e2-47f3-843b-8627c84e18df\" (UID: \"1320c907-c1e2-47f3-843b-8627c84e18df\") "
Sep 15 21:31:00.299772 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.299749    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-model-cache" (OuterVolumeSpecName: "model-cache") pod "1320c907-c1e2-47f3-843b-8627c84e18df" (UID: "1320c907-c1e2-47f3-843b-8627c84e18df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:00.299833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.299784    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-home" (OuterVolumeSpecName: "home") pod "1320c907-c1e2-47f3-843b-8627c84e18df" (UID: "1320c907-c1e2-47f3-843b-8627c84e18df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:00.299833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.299819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1320c907-c1e2-47f3-843b-8627c84e18df" (UID: "1320c907-c1e2-47f3-843b-8627c84e18df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:00.300047 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.300030    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.300116 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.300052    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.300116 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.300065    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.301614 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.301596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-dshm" (OuterVolumeSpecName: "dshm") pod "1320c907-c1e2-47f3-843b-8627c84e18df" (UID: "1320c907-c1e2-47f3-843b-8627c84e18df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:00.301691 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.301665    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1320c907-c1e2-47f3-843b-8627c84e18df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1320c907-c1e2-47f3-843b-8627c84e18df" (UID: "1320c907-c1e2-47f3-843b-8627c84e18df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:31:00.301738 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.301700    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1320c907-c1e2-47f3-843b-8627c84e18df-kube-api-access-lm8gs" (OuterVolumeSpecName: "kube-api-access-lm8gs") pod "1320c907-c1e2-47f3-843b-8627c84e18df" (UID: "1320c907-c1e2-47f3-843b-8627c84e18df"). InnerVolumeSpecName "kube-api-access-lm8gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:00.401300 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.401269    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1320c907-c1e2-47f3-843b-8627c84e18df-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.401300 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.401295    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8gs\" (UniqueName: \"kubernetes.io/projected/1320c907-c1e2-47f3-843b-8627c84e18df-kube-api-access-lm8gs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.401300 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.401306    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1320c907-c1e2-47f3-843b-8627c84e18df-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.484056 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.483978    2563 generic.go:356] "Generic (PLEG): container finished" podID="1320c907-c1e2-47f3-843b-8627c84e18df" containerID="b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402" exitCode=0
Sep 15 21:31:00.484056 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.484021    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"
Sep 15 21:31:00.484218 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.484065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7" event={"ID":"1320c907-c1e2-47f3-843b-8627c84e18df","Type":"ContainerDied","Data":"b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402"}
Sep 15 21:31:00.484218 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.484107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7" event={"ID":"1320c907-c1e2-47f3-843b-8627c84e18df","Type":"ContainerDied","Data":"8bdc9efc82bf84adc6a4ca59d69ce0ec519a2778c3c879dfbee4fea6394ffa88"}
Sep 15 21:31:00.484218 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.484124    2563 scope.go:117] "RemoveContainer" containerID="b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402"
Sep 15 21:31:00.494100 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.494090    2563 scope.go:117] "RemoveContainer" containerID="b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402"
Sep 15 21:31:00.494423 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:31:00.494404    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402\": container with ID starting with b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402 not found: ID does not exist" containerID="b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402"
Sep 15 21:31:00.494490 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.494432    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402"} err="failed to get container status \"b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402\": rpc error: code = NotFound desc = could not find container \"b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402\": container with ID starting with b46c56753a6bb98a455b7fa84b7d3c266f263ea69795ca080788176c24b8c402 not found: ID does not exist"
Sep 15 21:31:00.505693 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.505672    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"]
Sep 15 21:31:00.509587 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.509570    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cfc1f4/test-model-cfc1f4-kserve-5566c4566c-vbqt7"]
Sep 15 21:31:00.532963 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:00.532941    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1320c907-c1e2-47f3-843b-8627c84e18df" path="/var/lib/kubelet/pods/1320c907-c1e2-47f3-843b-8627c84e18df/volumes"
Sep 15 21:31:12.526222 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.526161    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5" podUID="808d4557-d2dd-40d1-be92-d605914669c8" containerName="main" containerID="cri-o://bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54" gracePeriod=45
Sep 15 21:31:12.768695 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.768673    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:31:12.925121 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925095    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808d4557-d2dd-40d1-be92-d605914669c8-tls-certs\") pod \"808d4557-d2dd-40d1-be92-d605914669c8\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") "
Sep 15 21:31:12.925267 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925126    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-model-cache\") pod \"808d4557-d2dd-40d1-be92-d605914669c8\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") "
Sep 15 21:31:12.925267 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925214    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-tmp-dir\") pod \"808d4557-d2dd-40d1-be92-d605914669c8\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") "
Sep 15 21:31:12.925267 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925245    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-dshm\") pod \"808d4557-d2dd-40d1-be92-d605914669c8\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") "
Sep 15 21:31:12.925398 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wksrb\" (UniqueName: \"kubernetes.io/projected/808d4557-d2dd-40d1-be92-d605914669c8-kube-api-access-wksrb\") pod \"808d4557-d2dd-40d1-be92-d605914669c8\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") "
Sep 15 21:31:12.925398 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-home\") pod \"808d4557-d2dd-40d1-be92-d605914669c8\" (UID: \"808d4557-d2dd-40d1-be92-d605914669c8\") "
Sep 15 21:31:12.925493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925463    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "808d4557-d2dd-40d1-be92-d605914669c8" (UID: "808d4557-d2dd-40d1-be92-d605914669c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:12.925550 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925500    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "808d4557-d2dd-40d1-be92-d605914669c8" (UID: "808d4557-d2dd-40d1-be92-d605914669c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:12.925682 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925661    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:12.925747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925694    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:12.925747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.925662    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-home" (OuterVolumeSpecName: "home") pod "808d4557-d2dd-40d1-be92-d605914669c8" (UID: "808d4557-d2dd-40d1-be92-d605914669c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:12.927436 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.927412    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808d4557-d2dd-40d1-be92-d605914669c8-kube-api-access-wksrb" (OuterVolumeSpecName: "kube-api-access-wksrb") pod "808d4557-d2dd-40d1-be92-d605914669c8" (UID: "808d4557-d2dd-40d1-be92-d605914669c8"). InnerVolumeSpecName "kube-api-access-wksrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:12.927612 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.927595    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/808d4557-d2dd-40d1-be92-d605914669c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "808d4557-d2dd-40d1-be92-d605914669c8" (UID: "808d4557-d2dd-40d1-be92-d605914669c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:31:12.927671 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:12.927606    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-dshm" (OuterVolumeSpecName: "dshm") pod "808d4557-d2dd-40d1-be92-d605914669c8" (UID: "808d4557-d2dd-40d1-be92-d605914669c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:13.026504 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.026483    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:13.026504 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.026504    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wksrb\" (UniqueName: \"kubernetes.io/projected/808d4557-d2dd-40d1-be92-d605914669c8-kube-api-access-wksrb\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:13.026630 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.026514    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808d4557-d2dd-40d1-be92-d605914669c8-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:13.026630 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.026523    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808d4557-d2dd-40d1-be92-d605914669c8-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:13.568205 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.568178    2563 generic.go:356] "Generic (PLEG): container finished" podID="808d4557-d2dd-40d1-be92-d605914669c8" containerID="bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54" exitCode=0
Sep 15 21:31:13.568599 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.568224    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"
Sep 15 21:31:13.568599 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.568261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5" event={"ID":"808d4557-d2dd-40d1-be92-d605914669c8","Type":"ContainerDied","Data":"bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54"}
Sep 15 21:31:13.568599 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.568306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5" event={"ID":"808d4557-d2dd-40d1-be92-d605914669c8","Type":"ContainerDied","Data":"a195a39a022c1c4fa4ffdcbbdc4bc19b9331233745e9055ddedfc67326993c6f"}
Sep 15 21:31:13.568599 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.568331    2563 scope.go:117] "RemoveContainer" containerID="bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54"
Sep 15 21:31:13.578236 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.578218    2563 scope.go:117] "RemoveContainer" containerID="bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54"
Sep 15 21:31:13.578514 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:31:13.578498    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54\": container with ID starting with bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54 not found: ID does not exist" containerID="bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54"
Sep 15 21:31:13.578577 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.578521    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54"} err="failed to get container status \"bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54\": rpc error: code = NotFound desc = could not find container \"bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54\": container with ID starting with bd6a5561aa1cfe6b2d5ce87909468684a7358536df534bf99d0d94094ce29e54 not found: ID does not exist"
Sep 15 21:31:13.589703 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.589675    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"]
Sep 15 21:31:13.594298 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:13.594279    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f0d7d5/noten-model-f0d7d5-kserve-5d5bbf7ff-mkkj5"]
Sep 15 21:31:14.527187 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:14.527147    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="808d4557-d2dd-40d1-be92-d605914669c8" path="/var/lib/kubelet/pods/808d4557-d2dd-40d1-be92-d605914669c8/volumes"
Sep 15 21:31:17.530422 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.530388    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"]
Sep 15 21:31:17.530851 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.530636    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2" podUID="5a41def5-0a96-49cb-95fd-fd46b7743ef3" containerName="maas-api" containerID="cri-o://2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3" gracePeriod=30
Sep 15 21:31:17.783901 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.783843    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:31:17.880394 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.880331    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c6p6\" (UniqueName: \"kubernetes.io/projected/5a41def5-0a96-49cb-95fd-fd46b7743ef3-kube-api-access-6c6p6\") pod \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") "
Sep 15 21:31:17.880549 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.880446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a41def5-0a96-49cb-95fd-fd46b7743ef3-openshift-service-ca\") pod \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") "
Sep 15 21:31:17.880549 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.880512    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-metrics-certs\") pod \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") "
Sep 15 21:31:17.880794 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.880557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-maas-api-tls\") pod \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\" (UID: \"5a41def5-0a96-49cb-95fd-fd46b7743ef3\") "
Sep 15 21:31:17.880868 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.880789    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a41def5-0a96-49cb-95fd-fd46b7743ef3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a41def5-0a96-49cb-95fd-fd46b7743ef3" (UID: "5a41def5-0a96-49cb-95fd-fd46b7743ef3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:17.880958 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.880942    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a41def5-0a96-49cb-95fd-fd46b7743ef3-openshift-service-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:17.882488 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.882466    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a41def5-0a96-49cb-95fd-fd46b7743ef3-kube-api-access-6c6p6" (OuterVolumeSpecName: "kube-api-access-6c6p6") pod "5a41def5-0a96-49cb-95fd-fd46b7743ef3" (UID: "5a41def5-0a96-49cb-95fd-fd46b7743ef3"). InnerVolumeSpecName "kube-api-access-6c6p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:17.882595 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.882576    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5a41def5-0a96-49cb-95fd-fd46b7743ef3" (UID: "5a41def5-0a96-49cb-95fd-fd46b7743ef3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:31:17.882655 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.882615    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a41def5-0a96-49cb-95fd-fd46b7743ef3" (UID: "5a41def5-0a96-49cb-95fd-fd46b7743ef3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:31:17.982495 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.982467    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6p6\" (UniqueName: \"kubernetes.io/projected/5a41def5-0a96-49cb-95fd-fd46b7743ef3-kube-api-access-6c6p6\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:17.982495 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.982490    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-metrics-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:17.982495 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:17.982500    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a41def5-0a96-49cb-95fd-fd46b7743ef3-maas-api-tls\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:18.599484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:18.599456    2563 generic.go:356] "Generic (PLEG): container finished" podID="5a41def5-0a96-49cb-95fd-fd46b7743ef3" containerID="2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3" exitCode=0
Sep 15 21:31:18.599820 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:18.599498    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"
Sep 15 21:31:18.599820 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:18.599514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2" event={"ID":"5a41def5-0a96-49cb-95fd-fd46b7743ef3","Type":"ContainerDied","Data":"2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3"}
Sep 15 21:31:18.599820 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:18.599571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2" event={"ID":"5a41def5-0a96-49cb-95fd-fd46b7743ef3","Type":"ContainerDied","Data":"5fb08607c147e09bea9b6a8394b03875a9717b0d66873f9b307629d432fd1941"}
Sep 15 21:31:18.599820 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:18.599599    2563 scope.go:117] "RemoveContainer" containerID="2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3"
Sep 15 21:31:18.609470 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:18.609450    2563 scope.go:117] "RemoveContainer" containerID="2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3"
Sep 15 21:31:18.609763 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:31:18.609736    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3\": container with ID starting with 2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3 not found: ID does not exist" containerID="2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3"
Sep 15 21:31:18.609823 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:18.609770    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3"} err="failed to get container status \"2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3\": rpc error: code = NotFound desc = could not find container \"2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3\": container with ID starting with 2d0008a0b86a1f6c75f592564ce2dcce9ca1a668fb582386ce8421a6298e80a3 not found: ID does not exist"
Sep 15 21:31:18.617204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:18.617184    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"]
Sep 15 21:31:18.620022 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:18.619998    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cfc1f4-8669bcd44b-5srz2"]
Sep 15 21:31:20.527395 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:20.527369    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a41def5-0a96-49cb-95fd-fd46b7743ef3" path="/var/lib/kubelet/pods/5a41def5-0a96-49cb-95fd-fd46b7743ef3/volumes"
Sep 15 21:31:23.507306 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507271    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"]
Sep 15 21:31:23.507837 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507819    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808d4557-d2dd-40d1-be92-d605914669c8" containerName="main"
Sep 15 21:31:23.507837 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507839    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="808d4557-d2dd-40d1-be92-d605914669c8" containerName="main"
Sep 15 21:31:23.507927 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507849    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7436e203-13f4-42e0-8f07-646371949a00" containerName="test-curl-7851-50dcaf"
Sep 15 21:31:23.507927 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507854    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7436e203-13f4-42e0-8f07-646371949a00" containerName="test-curl-7851-50dcaf"
Sep 15 21:31:23.507927 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507873    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a41def5-0a96-49cb-95fd-fd46b7743ef3" containerName="maas-api"
Sep 15 21:31:23.507927 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507879    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a41def5-0a96-49cb-95fd-fd46b7743ef3" containerName="maas-api"
Sep 15 21:31:23.507927 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507887    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1320c907-c1e2-47f3-843b-8627c84e18df" containerName="main"
Sep 15 21:31:23.507927 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507893    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1320c907-c1e2-47f3-843b-8627c84e18df" containerName="main"
Sep 15 21:31:23.508119 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507967    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="808d4557-d2dd-40d1-be92-d605914669c8" containerName="main"
Sep 15 21:31:23.508119 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507975    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a41def5-0a96-49cb-95fd-fd46b7743ef3" containerName="maas-api"
Sep 15 21:31:23.508119 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507981    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7436e203-13f4-42e0-8f07-646371949a00" containerName="test-curl-7851-50dcaf"
Sep 15 21:31:23.508119 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.507990    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1320c907-c1e2-47f3-843b-8627c84e18df" containerName="main"
Sep 15 21:31:23.511148 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.511131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.513240 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.513219    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-318c22\"/\"distinct-w0-318c22-kserve-self-signed-certs\""
Sep 15 21:31:23.520667 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.520646    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"]
Sep 15 21:31:23.630053 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.630032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-home\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.630194 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.630058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f170c4b-dff1-44fc-8081-4a7785446f59-tls-certs\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.630194 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.630080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-dshm\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.630194 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.630172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-model-cache\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.630380 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.630265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpcv8\" (UniqueName: \"kubernetes.io/projected/3f170c4b-dff1-44fc-8081-4a7785446f59-kube-api-access-zpcv8\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.630380 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.630323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-tmp-dir\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.731733 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.731706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-dshm\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.731853 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.731756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-model-cache\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.731853 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.731794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpcv8\" (UniqueName: \"kubernetes.io/projected/3f170c4b-dff1-44fc-8081-4a7785446f59-kube-api-access-zpcv8\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.731853 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.731825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-tmp-dir\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.731965 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.731865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-home\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.731965 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.731881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f170c4b-dff1-44fc-8081-4a7785446f59-tls-certs\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.732229 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.732164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-model-cache\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.732310 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.732224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-home\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.732310 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.732285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-tmp-dir\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.733897 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.733875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-dshm\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.734161 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.734145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f170c4b-dff1-44fc-8081-4a7785446f59-tls-certs\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.739099 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.739082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpcv8\" (UniqueName: \"kubernetes.io/projected/3f170c4b-dff1-44fc-8081-4a7785446f59-kube-api-access-zpcv8\") pod \"distinct-w0-318c22-kserve-c4cbcf446-2t47n\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:23.821901 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:23.821837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:24.089989 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:24.089959    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"]
Sep 15 21:31:24.090464 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:31:24.090444    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f170c4b_dff1_44fc_8081_4a7785446f59.slice/crio-a7ffff45a2c5a8fd63f919dcbcdd7da912c35eac41579182ea46f450151eb75a WatchSource:0}: Error finding container a7ffff45a2c5a8fd63f919dcbcdd7da912c35eac41579182ea46f450151eb75a: Status 404 returned error can't find the container with id a7ffff45a2c5a8fd63f919dcbcdd7da912c35eac41579182ea46f450151eb75a
Sep 15 21:31:24.638430 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:24.638394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n" event={"ID":"3f170c4b-dff1-44fc-8081-4a7785446f59","Type":"ContainerStarted","Data":"57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871"}
Sep 15 21:31:24.638430 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:24.638432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n" event={"ID":"3f170c4b-dff1-44fc-8081-4a7785446f59","Type":"ContainerStarted","Data":"a7ffff45a2c5a8fd63f919dcbcdd7da912c35eac41579182ea46f450151eb75a"}
Sep 15 21:31:24.655468 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:24.655421    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n" podStartSLOduration=1.655405339 podStartE2EDuration="1.655405339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:24.653755662 +0000 UTC m=+1274.616813044" watchObservedRunningTime="2026-09-15 21:31:24.655405339 +0000 UTC m=+1274.618462731"
Sep 15 21:31:27.697039 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:27.697010    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"]
Sep 15 21:31:33.822396 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:33.822365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:33.822952 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:33.822429    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:33.834466 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:33.834444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:34.707918 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:34.707885    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:31:40.296103 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.296069    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"]
Sep 15 21:31:40.299381 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.299337    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.301265 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.301243    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"e2e-conflict-b-8487c1aa-kserve-self-signed-certs\""
Sep 15 21:31:40.307202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.307004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"]
Sep 15 21:31:40.386453 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.386427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-home\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.386577 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.386511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jht5\" (UniqueName: \"kubernetes.io/projected/78994622-1536-4237-b598-f3e2e518386c-kube-api-access-8jht5\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.386627 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.386589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-dshm\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.386663 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.386644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-model-cache\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.386697 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.386668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-tmp-dir\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.386697 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.386690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78994622-1536-4237-b598-f3e2e518386c-tls-certs\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.488405 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.488373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-dshm\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.488569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.488424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-model-cache\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.488569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.488443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-tmp-dir\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.488569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.488465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78994622-1536-4237-b598-f3e2e518386c-tls-certs\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.488569 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.488543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-home\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.488800 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.488620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jht5\" (UniqueName: \"kubernetes.io/projected/78994622-1536-4237-b598-f3e2e518386c-kube-api-access-8jht5\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.488871 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.488849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-tmp-dir\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.488925 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.488883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-home\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.488979 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.488959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-model-cache\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.490512 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.490486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-dshm\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.490763 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.490747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78994622-1536-4237-b598-f3e2e518386c-tls-certs\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.497562 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.497542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jht5\" (UniqueName: \"kubernetes.io/projected/78994622-1536-4237-b598-f3e2e518386c-kube-api-access-8jht5\") pod \"e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.609868 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.609799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:40.883001 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:40.882973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"]
Sep 15 21:31:40.883427 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:31:40.883398    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78994622_1536_4237_b598_f3e2e518386c.slice/crio-52bf5cac10437649912b5ea095410b2cc895b1afd08690972edd18487a90711b WatchSource:0}: Error finding container 52bf5cac10437649912b5ea095410b2cc895b1afd08690972edd18487a90711b: Status 404 returned error can't find the container with id 52bf5cac10437649912b5ea095410b2cc895b1afd08690972edd18487a90711b
Sep 15 21:31:41.747655 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:41.747611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf" event={"ID":"78994622-1536-4237-b598-f3e2e518386c","Type":"ContainerStarted","Data":"6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53"}
Sep 15 21:31:41.748129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:41.747667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf" event={"ID":"78994622-1536-4237-b598-f3e2e518386c","Type":"ContainerStarted","Data":"52bf5cac10437649912b5ea095410b2cc895b1afd08690972edd18487a90711b"}
Sep 15 21:31:41.768295 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:41.768223    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf" podStartSLOduration=1.768199888 podStartE2EDuration="1.768199888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:41.765469398 +0000 UTC m=+1291.728526781" watchObservedRunningTime="2026-09-15 21:31:41.768199888 +0000 UTC m=+1291.731257271"
Sep 15 21:31:42.716330 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:42.716295    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6" podUID="9c469b68-d775-46f7-b693-466de65efc08" containerName="main" containerID="cri-o://d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a" gracePeriod=45
Sep 15 21:31:42.974556 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:42.974492    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:31:43.119029 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.118996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c469b68-d775-46f7-b693-466de65efc08-tls-certs\") pod \"9c469b68-d775-46f7-b693-466de65efc08\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") "
Sep 15 21:31:43.119206 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119041    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-tmp-dir\") pod \"9c469b68-d775-46f7-b693-466de65efc08\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") "
Sep 15 21:31:43.119206 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119079    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-dshm\") pod \"9c469b68-d775-46f7-b693-466de65efc08\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") "
Sep 15 21:31:43.119206 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svmq2\" (UniqueName: \"kubernetes.io/projected/9c469b68-d775-46f7-b693-466de65efc08-kube-api-access-svmq2\") pod \"9c469b68-d775-46f7-b693-466de65efc08\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") "
Sep 15 21:31:43.119374 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119214    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-model-cache\") pod \"9c469b68-d775-46f7-b693-466de65efc08\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") "
Sep 15 21:31:43.119374 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119265    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-home\") pod \"9c469b68-d775-46f7-b693-466de65efc08\" (UID: \"9c469b68-d775-46f7-b693-466de65efc08\") "
Sep 15 21:31:43.119374 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c469b68-d775-46f7-b693-466de65efc08" (UID: "9c469b68-d775-46f7-b693-466de65efc08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:43.119599 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119568    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c469b68-d775-46f7-b693-466de65efc08" (UID: "9c469b68-d775-46f7-b693-466de65efc08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:43.119719 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119570    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-home" (OuterVolumeSpecName: "home") pod "9c469b68-d775-46f7-b693-466de65efc08" (UID: "9c469b68-d775-46f7-b693-466de65efc08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:43.119903 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119885    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:43.119903 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119904    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:43.120050 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.119914    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:43.121401 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.121335    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-dshm" (OuterVolumeSpecName: "dshm") pod "9c469b68-d775-46f7-b693-466de65efc08" (UID: "9c469b68-d775-46f7-b693-466de65efc08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:43.121478 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.121337    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c469b68-d775-46f7-b693-466de65efc08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c469b68-d775-46f7-b693-466de65efc08" (UID: "9c469b68-d775-46f7-b693-466de65efc08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:31:43.121478 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.121405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c469b68-d775-46f7-b693-466de65efc08-kube-api-access-svmq2" (OuterVolumeSpecName: "kube-api-access-svmq2") pod "9c469b68-d775-46f7-b693-466de65efc08" (UID: "9c469b68-d775-46f7-b693-466de65efc08"). InnerVolumeSpecName "kube-api-access-svmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:43.220635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.220599    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c469b68-d775-46f7-b693-466de65efc08-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:43.220635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.220628    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c469b68-d775-46f7-b693-466de65efc08-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:43.220635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.220638    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svmq2\" (UniqueName: \"kubernetes.io/projected/9c469b68-d775-46f7-b693-466de65efc08-kube-api-access-svmq2\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:43.761496 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.761464    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c469b68-d775-46f7-b693-466de65efc08" containerID="d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a" exitCode=0
Sep 15 21:31:43.761665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.761507    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"
Sep 15 21:31:43.761665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.761553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6" event={"ID":"9c469b68-d775-46f7-b693-466de65efc08","Type":"ContainerDied","Data":"d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a"}
Sep 15 21:31:43.761665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.761589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6" event={"ID":"9c469b68-d775-46f7-b693-466de65efc08","Type":"ContainerDied","Data":"c714798bc8ed7d7818551084d81412715481f680c57326c55408dac8215c7f55"}
Sep 15 21:31:43.761665 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.761626    2563 scope.go:117] "RemoveContainer" containerID="d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a"
Sep 15 21:31:43.771814 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.771796    2563 scope.go:117] "RemoveContainer" containerID="d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a"
Sep 15 21:31:43.772104 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:31:43.772086    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a\": container with ID starting with d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a not found: ID does not exist" containerID="d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a"
Sep 15 21:31:43.772167 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.772117    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a"} err="failed to get container status \"d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a\": rpc error: code = NotFound desc = could not find container \"d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a\": container with ID starting with d8a41f79393ffc2de4d35133462ebe9575454eee1d80842a08def2ee497bcb2a not found: ID does not exist"
Sep 15 21:31:43.801778 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.801745    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"]
Sep 15 21:31:43.817163 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:43.817129    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f457e5/test-model-f457e5-kserve-5b9c7994bf-2wtr6"]
Sep 15 21:31:44.526943 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:44.526914    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c469b68-d775-46f7-b693-466de65efc08" path="/var/lib/kubelet/pods/9c469b68-d775-46f7-b693-466de65efc08/volumes"
Sep 15 21:31:47.071527 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:47.071496    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"]
Sep 15 21:31:47.072011 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:47.071739    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy" containerID="cri-o://86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a" gracePeriod=30
Sep 15 21:31:48.997657 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:48.997619    2563 patch_prober.go:28] interesting pod/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:31:48.998018 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:48.997678    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:31:50.609891 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:50.609858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:50.610273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:50.609905    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:50.622477 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:50.622450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:50.815690 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:50.815662    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:31:52.323003 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.322977    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:31:52.424049 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istiod-ca-cert\") pod \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") "
Sep 15 21:31:52.424251 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424073    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-envoy\") pod \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") "
Sep 15 21:31:52.424251 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-token\") pod \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") "
Sep 15 21:31:52.424251 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-socket\") pod \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") "
Sep 15 21:31:52.424251 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424144    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-credential-socket\") pod \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") "
Sep 15 21:31:52.424251 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424164    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-certs\") pod \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") "
Sep 15 21:31:52.424251 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424207    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-data\") pod \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") "
Sep 15 21:31:52.424607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424306    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjtmx\" (UniqueName: \"kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-kube-api-access-wjtmx\") pod \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") "
Sep 15 21:31:52.424607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-podinfo\") pod \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\" (UID: \"375b66c1-4e10-4dc4-8315-3ec6be3868a2\") "
Sep 15 21:31:52.424607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424513    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "375b66c1-4e10-4dc4-8315-3ec6be3868a2" (UID: "375b66c1-4e10-4dc4-8315-3ec6be3868a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:52.424607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424513    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "375b66c1-4e10-4dc4-8315-3ec6be3868a2" (UID: "375b66c1-4e10-4dc4-8315-3ec6be3868a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:52.424607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "375b66c1-4e10-4dc4-8315-3ec6be3868a2" (UID: "375b66c1-4e10-4dc4-8315-3ec6be3868a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:52.424607 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424588    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "375b66c1-4e10-4dc4-8315-3ec6be3868a2" (UID: "375b66c1-4e10-4dc4-8315-3ec6be3868a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:52.424900 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424714    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "375b66c1-4e10-4dc4-8315-3ec6be3868a2" (UID: "375b66c1-4e10-4dc4-8315-3ec6be3868a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:52.424900 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424857    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istiod-ca-cert\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:52.424900 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424879    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-socket\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:52.424900 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424892    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-credential-socket\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:52.425042 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424905    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-workload-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:52.425042 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.424917    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-data\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:52.426605 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.426581    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-kube-api-access-wjtmx" (OuterVolumeSpecName: "kube-api-access-wjtmx") pod "375b66c1-4e10-4dc4-8315-3ec6be3868a2" (UID: "375b66c1-4e10-4dc4-8315-3ec6be3868a2"). InnerVolumeSpecName "kube-api-access-wjtmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:52.426714 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.426600    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "375b66c1-4e10-4dc4-8315-3ec6be3868a2" (UID: "375b66c1-4e10-4dc4-8315-3ec6be3868a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:52.426778 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.426758    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "375b66c1-4e10-4dc4-8315-3ec6be3868a2" (UID: "375b66c1-4e10-4dc4-8315-3ec6be3868a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:52.426819 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.426790    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "375b66c1-4e10-4dc4-8315-3ec6be3868a2" (UID: "375b66c1-4e10-4dc4-8315-3ec6be3868a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:31:52.525614 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.525580    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjtmx\" (UniqueName: \"kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-kube-api-access-wjtmx\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:52.525614 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.525603    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-podinfo\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:52.525614 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.525612    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-envoy\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:52.525614 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.525622    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375b66c1-4e10-4dc4-8315-3ec6be3868a2-istio-token\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:31:52.819098 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.819005    2563 generic.go:356] "Generic (PLEG): container finished" podID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerID="86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a" exitCode=0
Sep 15 21:31:52.819098 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.819041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" event={"ID":"375b66c1-4e10-4dc4-8315-3ec6be3868a2","Type":"ContainerDied","Data":"86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a"}
Sep 15 21:31:52.819098 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.819095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f" event={"ID":"375b66c1-4e10-4dc4-8315-3ec6be3868a2","Type":"ContainerDied","Data":"7d1243c44c02b5722fd6b8b36c6efc7515c62ed1087db4ebea0d6080b2b39e90"}
Sep 15 21:31:52.819327 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.819118    2563 scope.go:117] "RemoveContainer" containerID="86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a"
Sep 15 21:31:52.819327 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.819050    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"
Sep 15 21:31:52.828837 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.828811    2563 scope.go:117] "RemoveContainer" containerID="e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6"
Sep 15 21:31:52.836846 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.836822    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"]
Sep 15 21:31:52.838104 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.838088    2563 scope.go:117] "RemoveContainer" containerID="86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a"
Sep 15 21:31:52.838390 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:31:52.838374    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a\": container with ID starting with 86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a not found: ID does not exist" containerID="86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a"
Sep 15 21:31:52.838453 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.838398    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a"} err="failed to get container status \"86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a\": rpc error: code = NotFound desc = could not find container \"86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a\": container with ID starting with 86c13187386481c0b98c44977316b565e0dd3ba68ba33ef05d1b545e283b817a not found: ID does not exist"
Sep 15 21:31:52.838453 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.838416    2563 scope.go:117] "RemoveContainer" containerID="e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6"
Sep 15 21:31:52.838641 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:31:52.838626    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6\": container with ID starting with e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6 not found: ID does not exist" containerID="e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6"
Sep 15 21:31:52.838688 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.838647    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6"} err="failed to get container status \"e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6\": rpc error: code = NotFound desc = could not find container \"e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6\": container with ID starting with e69081c1add73a211700a3001ba309b0a249338a0a98ffa7c0e15989cc3d08a6 not found: ID does not exist"
Sep 15 21:31:52.842726 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:52.842705    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f457e5-openshift-default-585fc6bbb4-lcq6f"]
Sep 15 21:31:54.526695 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.526662    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" path="/var/lib/kubelet/pods/375b66c1-4e10-4dc4-8315-3ec6be3868a2/volumes"
Sep 15 21:31:54.913585 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.913554    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"]
Sep 15 21:31:54.913979 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.913967    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c469b68-d775-46f7-b693-466de65efc08" containerName="main"
Sep 15 21:31:54.913979 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.913980    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c469b68-d775-46f7-b693-466de65efc08" containerName="main"
Sep 15 21:31:54.914060 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.914004    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy"
Sep 15 21:31:54.914060 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.914010    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy"
Sep 15 21:31:54.914060 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.914024    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy"
Sep 15 21:31:54.914060 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.914030    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy"
Sep 15 21:31:54.914180 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.914089    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy"
Sep 15 21:31:54.914180 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.914099    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c469b68-d775-46f7-b693-466de65efc08" containerName="main"
Sep 15 21:31:54.918592 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.918565    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:54.920455 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.920435    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-f36399-serving-cert\""
Sep 15 21:31:54.928096 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.928074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"]
Sep 15 21:31:54.948236 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.948201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:54.948444 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.948244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38926e29-d812-4502-89ba-3a7a01b6f54f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:54.948444 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.948393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpbq\" (UniqueName: \"kubernetes.io/projected/38926e29-d812-4502-89ba-3a7a01b6f54f-kube-api-access-xkpbq\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:54.948444 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:54.948424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:55.049304 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:55.049270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpbq\" (UniqueName: \"kubernetes.io/projected/38926e29-d812-4502-89ba-3a7a01b6f54f-kube-api-access-xkpbq\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:55.049528 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:55.049311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:55.049528 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:55.049400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:55.049528 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:55.049421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38926e29-d812-4502-89ba-3a7a01b6f54f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:55.049528 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:31:55.049464    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-serving-cert: secret "maas-api-e2e-shared-a-w3-f36399-serving-cert" not found
Sep 15 21:31:55.049701 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:31:55.049540    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-maas-api-tls podName:38926e29-d812-4502-89ba-3a7a01b6f54f nodeName:}" failed. No retries permitted until 2026-09-15 21:31:55.549517834 +0000 UTC m=+1305.512575207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-maas-api-tls") pod "maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k" (UID: "38926e29-d812-4502-89ba-3a7a01b6f54f") : secret "maas-api-e2e-shared-a-w3-f36399-serving-cert" not found
Sep 15 21:31:55.050094 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:55.050062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38926e29-d812-4502-89ba-3a7a01b6f54f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:55.051886 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:55.051863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:55.057499 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:55.057481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpbq\" (UniqueName: \"kubernetes.io/projected/38926e29-d812-4502-89ba-3a7a01b6f54f-kube-api-access-xkpbq\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:55.555107 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:55.555072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:55.557338 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:55.557319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:55.829757 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:55.829667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:56.106454 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:56.106424    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"]
Sep 15 21:31:56.108095 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:31:56.108070    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38926e29_d812_4502_89ba_3a7a01b6f54f.slice/crio-1387fb12d3800196e4358b2884da4c0cd8a352fa1ca235921574da6b872ac4ff WatchSource:0}: Error finding container 1387fb12d3800196e4358b2884da4c0cd8a352fa1ca235921574da6b872ac4ff: Status 404 returned error can't find the container with id 1387fb12d3800196e4358b2884da4c0cd8a352fa1ca235921574da6b872ac4ff
Sep 15 21:31:56.851019 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:56.850911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k" event={"ID":"38926e29-d812-4502-89ba-3a7a01b6f54f","Type":"ContainerStarted","Data":"1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b"}
Sep 15 21:31:56.851019 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:56.850969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k" event={"ID":"38926e29-d812-4502-89ba-3a7a01b6f54f","Type":"ContainerStarted","Data":"1387fb12d3800196e4358b2884da4c0cd8a352fa1ca235921574da6b872ac4ff"}
Sep 15 21:31:56.851019 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:56.850995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:31:56.867548 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:31:56.867486    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k" podStartSLOduration=2.514792217 podStartE2EDuration="2.867466714s" totalImagesPullingTime="352.674497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:56.865992724 +0000 UTC m=+1306.829050105" watchObservedRunningTime="2026-09-15 21:31:56.867466714 +0000 UTC m=+1306.830524096"
Sep 15 21:32:02.864109 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:02.864071    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:32:19.113975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.113942    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"]
Sep 15 21:32:19.114528 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.114512    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="375b66c1-4e10-4dc4-8315-3ec6be3868a2" containerName="istio-proxy"
Sep 15 21:32:19.119177 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.119156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.121476 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.121456    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d1980a0d\"/\"openshift-service-ca.crt\""
Sep 15 21:32:19.122271 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.122252    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d1980a0d\"/\"e2e-lifecycle-model-d1980a0d-kserve-self-signed-certs\""
Sep 15 21:32:19.122369 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.122325    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d1980a0d\"/\"kube-root-ca.crt\""
Sep 15 21:32:19.122449 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.122375    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d1980a0d\"/\"default-dockercfg-bvj5f\""
Sep 15 21:32:19.127899 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.127875    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"]
Sep 15 21:32:19.308669 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.308633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-model-cache\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.308669 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.308667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rmn\" (UniqueName: \"kubernetes.io/projected/19362e19-23c2-4e4b-b7bf-a0d439bb609d-kube-api-access-s6rmn\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.308888 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.308690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tls-certs\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.308888 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.308779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-home\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.308888 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.308813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tmp-dir\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.308888 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.308870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-dshm\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.409761 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.409720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-home\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.409761 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.409766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tmp-dir\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.410013 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.409816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-dshm\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.410013 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.409878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-model-cache\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.410013 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.409900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rmn\" (UniqueName: \"kubernetes.io/projected/19362e19-23c2-4e4b-b7bf-a0d439bb609d-kube-api-access-s6rmn\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.410013 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.409917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tls-certs\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.410214 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.410175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-home\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.410280 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.410205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tmp-dir\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.410280 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.410243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-model-cache\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.412117 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.412101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-dshm\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.412337 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.412321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tls-certs\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.420060 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.420037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rmn\" (UniqueName: \"kubernetes.io/projected/19362e19-23c2-4e4b-b7bf-a0d439bb609d-kube-api-access-s6rmn\") pod \"e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") " pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.431042 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.431015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:19.705868 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:19.705835    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"]
Sep 15 21:32:19.706931 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:32:19.706909    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19362e19_23c2_4e4b_b7bf_a0d439bb609d.slice/crio-9479ced6693d93bee7569505863fafc19f59ff8d7c897491bd80bfcd17764508 WatchSource:0}: Error finding container 9479ced6693d93bee7569505863fafc19f59ff8d7c897491bd80bfcd17764508: Status 404 returned error can't find the container with id 9479ced6693d93bee7569505863fafc19f59ff8d7c897491bd80bfcd17764508
Sep 15 21:32:20.005333 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:20.005237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn" event={"ID":"19362e19-23c2-4e4b-b7bf-a0d439bb609d","Type":"ContainerStarted","Data":"bfbbfc85bd42e419ea41197b525f7bad1141b7c018fb4e5c8d6cbf8f54fff16e"}
Sep 15 21:32:20.005333 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:20.005279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn" event={"ID":"19362e19-23c2-4e4b-b7bf-a0d439bb609d","Type":"ContainerStarted","Data":"9479ced6693d93bee7569505863fafc19f59ff8d7c897491bd80bfcd17764508"}
Sep 15 21:32:20.020602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:20.020552    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn" podStartSLOduration=1.02053353 podStartE2EDuration="1.02053353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:20.019685365 +0000 UTC m=+1329.982742750" watchObservedRunningTime="2026-09-15 21:32:20.02053353 +0000 UTC m=+1329.983590925"
Sep 15 21:32:29.431307 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:29.431263    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:29.431846 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:29.431333    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:29.443786 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:29.443758    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:32:30.087017 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:32:30.086980    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:33:09.107484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.107447    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"]
Sep 15 21:33:09.111258 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.111239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.113311 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.113290    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-318c22\"/\"distinct-2-w0-318c22-kserve-self-signed-certs\""
Sep 15 21:33:09.118960 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.118930    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"]
Sep 15 21:33:09.224644 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.224601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-home\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.224866 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.224662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tmp-dir\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.224866 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.224736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-dshm\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.224866 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.224807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-model-cache\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.224866 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.224838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tls-certs\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.225085 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.224873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbvg\" (UniqueName: \"kubernetes.io/projected/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-kube-api-access-mhbvg\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.325602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.325567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-home\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.325773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.325621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tmp-dir\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.325773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.325657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-dshm\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.325773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.325722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-model-cache\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.325773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.325742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tls-certs\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.325972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.325775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbvg\" (UniqueName: \"kubernetes.io/projected/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-kube-api-access-mhbvg\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.326051 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.326022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-home\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.326277 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.326256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-model-cache\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.326411 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.326288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tmp-dir\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.328015 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.327985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-dshm\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.328256 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.328240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tls-certs\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.333890 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.333870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbvg\" (UniqueName: \"kubernetes.io/projected/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-kube-api-access-mhbvg\") pod \"distinct-2-w0-318c22-kserve-576b7777f7-gk4jv\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") " pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.422439 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.422406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:09.693781 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:09.693751    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"]
Sep 15 21:33:09.694307 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:33:09.694270    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0c7e985_13f4_44f6_ada0_8aae1c5cedcb.slice/crio-6e961f31fe4f8c157980606ac7c50f8e151d5d6d1e3656d67e1f12f203add370 WatchSource:0}: Error finding container 6e961f31fe4f8c157980606ac7c50f8e151d5d6d1e3656d67e1f12f203add370: Status 404 returned error can't find the container with id 6e961f31fe4f8c157980606ac7c50f8e151d5d6d1e3656d67e1f12f203add370
Sep 15 21:33:10.336235 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:10.336196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv" event={"ID":"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb","Type":"ContainerStarted","Data":"cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db"}
Sep 15 21:33:10.336635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:10.336247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv" event={"ID":"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb","Type":"ContainerStarted","Data":"6e961f31fe4f8c157980606ac7c50f8e151d5d6d1e3656d67e1f12f203add370"}
Sep 15 21:33:10.355807 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:10.355747    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv" podStartSLOduration=1.355730272 podStartE2EDuration="1.355730272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:10.352904502 +0000 UTC m=+1380.315961877" watchObservedRunningTime="2026-09-15 21:33:10.355730272 +0000 UTC m=+1380.318787651"
Sep 15 21:33:11.588954 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:11.588918    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"]
Sep 15 21:33:19.423184 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:19.423143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:19.423726 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:19.423208    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:19.436365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:19.436326    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:20.408254 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:20.408226    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:33:26.608931 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:26.608892    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf" podUID="78994622-1536-4237-b598-f3e2e518386c" containerName="main" containerID="cri-o://6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53" gracePeriod=45
Sep 15 21:33:26.857971 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:26.857944    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:33:27.012926 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.012887    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jht5\" (UniqueName: \"kubernetes.io/projected/78994622-1536-4237-b598-f3e2e518386c-kube-api-access-8jht5\") pod \"78994622-1536-4237-b598-f3e2e518386c\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") "
Sep 15 21:33:27.013117 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-home\") pod \"78994622-1536-4237-b598-f3e2e518386c\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") "
Sep 15 21:33:27.013117 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-model-cache\") pod \"78994622-1536-4237-b598-f3e2e518386c\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") "
Sep 15 21:33:27.013218 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-dshm\") pod \"78994622-1536-4237-b598-f3e2e518386c\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") "
Sep 15 21:33:27.013218 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-tmp-dir\") pod \"78994622-1536-4237-b598-f3e2e518386c\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") "
Sep 15 21:33:27.013218 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013196    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78994622-1536-4237-b598-f3e2e518386c-tls-certs\") pod \"78994622-1536-4237-b598-f3e2e518386c\" (UID: \"78994622-1536-4237-b598-f3e2e518386c\") "
Sep 15 21:33:27.013400 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-home" (OuterVolumeSpecName: "home") pod "78994622-1536-4237-b598-f3e2e518386c" (UID: "78994622-1536-4237-b598-f3e2e518386c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:33:27.013400 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013335    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-model-cache" (OuterVolumeSpecName: "model-cache") pod "78994622-1536-4237-b598-f3e2e518386c" (UID: "78994622-1536-4237-b598-f3e2e518386c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:33:27.013524 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013461    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78994622-1536-4237-b598-f3e2e518386c" (UID: "78994622-1536-4237-b598-f3e2e518386c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:33:27.013635 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013619    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:27.013696 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013641    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:27.013696 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.013654    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:27.015548 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.015510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78994622-1536-4237-b598-f3e2e518386c-kube-api-access-8jht5" (OuterVolumeSpecName: "kube-api-access-8jht5") pod "78994622-1536-4237-b598-f3e2e518386c" (UID: "78994622-1536-4237-b598-f3e2e518386c"). InnerVolumeSpecName "kube-api-access-8jht5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:33:27.015684 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.015617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78994622-1536-4237-b598-f3e2e518386c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78994622-1536-4237-b598-f3e2e518386c" (UID: "78994622-1536-4237-b598-f3e2e518386c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:33:27.015684 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.015617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-dshm" (OuterVolumeSpecName: "dshm") pod "78994622-1536-4237-b598-f3e2e518386c" (UID: "78994622-1536-4237-b598-f3e2e518386c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:33:27.115026 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.114994    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78994622-1536-4237-b598-f3e2e518386c-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:27.115026 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.115022    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78994622-1536-4237-b598-f3e2e518386c-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:27.115026 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.115033    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jht5\" (UniqueName: \"kubernetes.io/projected/78994622-1536-4237-b598-f3e2e518386c-kube-api-access-8jht5\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:27.440834 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.440803    2563 generic.go:356] "Generic (PLEG): container finished" podID="78994622-1536-4237-b598-f3e2e518386c" containerID="6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53" exitCode=0
Sep 15 21:33:27.441034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.440855    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"
Sep 15 21:33:27.441034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.440886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf" event={"ID":"78994622-1536-4237-b598-f3e2e518386c","Type":"ContainerDied","Data":"6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53"}
Sep 15 21:33:27.441034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.440941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf" event={"ID":"78994622-1536-4237-b598-f3e2e518386c","Type":"ContainerDied","Data":"52bf5cac10437649912b5ea095410b2cc895b1afd08690972edd18487a90711b"}
Sep 15 21:33:27.441034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.440963    2563 scope.go:117] "RemoveContainer" containerID="6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53"
Sep 15 21:33:27.451005 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.450982    2563 scope.go:117] "RemoveContainer" containerID="6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53"
Sep 15 21:33:27.451304 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:33:27.451285    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53\": container with ID starting with 6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53 not found: ID does not exist" containerID="6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53"
Sep 15 21:33:27.451421 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.451314    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53"} err="failed to get container status \"6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53\": rpc error: code = NotFound desc = could not find container \"6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53\": container with ID starting with 6a7750813dca9fbb2c403f95cdb459a107c71175ffc5f182b2ffe25aa511fe53 not found: ID does not exist"
Sep 15 21:33:27.465696 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.465675    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"]
Sep 15 21:33:27.469319 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:27.469296    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-b-8487c1aa-kserve-5ccf8d8ff8-n8fcf"]
Sep 15 21:33:28.527558 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.527525    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78994622-1536-4237-b598-f3e2e518386c" path="/var/lib/kubelet/pods/78994622-1536-4237-b598-f3e2e518386c/volumes"
Sep 15 21:33:28.703894 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.703861    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"]
Sep 15 21:33:28.704539 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.704516    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78994622-1536-4237-b598-f3e2e518386c" containerName="main"
Sep 15 21:33:28.704539 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.704538    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="78994622-1536-4237-b598-f3e2e518386c" containerName="main"
Sep 15 21:33:28.704721 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.704693    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="78994622-1536-4237-b598-f3e2e518386c" containerName="main"
Sep 15 21:33:28.709452 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.709433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.712499 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.712476    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"default-dockercfg-cmswh\""
Sep 15 21:33:28.712634 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.712613    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"kube-root-ca.crt\""
Sep 15 21:33:28.712875 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.712856    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"openshift-service-ca.crt\""
Sep 15 21:33:28.712957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.712886    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"auth-test-model-f36399-kserve-self-signed-certs\""
Sep 15 21:33:28.716520 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.716501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"]
Sep 15 21:33:28.834182 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.834114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tmp-dir\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.834182 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.834159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrvp\" (UniqueName: \"kubernetes.io/projected/4e56b4af-b11d-4774-962c-3f7a9f4523e5-kube-api-access-ggrvp\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.834369 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.834264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-home\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.834369 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.834293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tls-certs\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.834369 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.834318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-model-cache\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.834489 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.834419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-dshm\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.934833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.934808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tmp-dir\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.934946 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.934849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrvp\" (UniqueName: \"kubernetes.io/projected/4e56b4af-b11d-4774-962c-3f7a9f4523e5-kube-api-access-ggrvp\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.934946 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.934901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-home\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.934946 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.934919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tls-certs\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.934946 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.934938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-model-cache\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.935100 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.935006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-dshm\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.935243 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.935221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tmp-dir\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.935281 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.935231    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-home\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.935480 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.935461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-model-cache\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.937213 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.937181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-dshm\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.937423 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.937409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tls-certs\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:28.943032 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:28.943015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrvp\" (UniqueName: \"kubernetes.io/projected/4e56b4af-b11d-4774-962c-3f7a9f4523e5-kube-api-access-ggrvp\") pod \"auth-test-model-f36399-kserve-98856b67f-ththb\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") " pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:29.020498 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:29.020474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:29.290923 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:29.290897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"]
Sep 15 21:33:29.292225 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:33:29.292200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e56b4af_b11d_4774_962c_3f7a9f4523e5.slice/crio-9aea52e5876bdd8920a4aa7c5c447882eea242b3378853ddff549ba2f36b2a3b WatchSource:0}: Error finding container 9aea52e5876bdd8920a4aa7c5c447882eea242b3378853ddff549ba2f36b2a3b: Status 404 returned error can't find the container with id 9aea52e5876bdd8920a4aa7c5c447882eea242b3378853ddff549ba2f36b2a3b
Sep 15 21:33:29.454845 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:29.454809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb" event={"ID":"4e56b4af-b11d-4774-962c-3f7a9f4523e5","Type":"ContainerStarted","Data":"bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9"}
Sep 15 21:33:29.454845 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:29.454846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb" event={"ID":"4e56b4af-b11d-4774-962c-3f7a9f4523e5","Type":"ContainerStarted","Data":"9aea52e5876bdd8920a4aa7c5c447882eea242b3378853ddff549ba2f36b2a3b"}
Sep 15 21:33:29.472063 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:29.472019    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb" podStartSLOduration=1.472004252 podStartE2EDuration="1.472004252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:29.469932597 +0000 UTC m=+1399.432989977" watchObservedRunningTime="2026-09-15 21:33:29.472004252 +0000 UTC m=+1399.435061774"
Sep 15 21:33:39.021277 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:39.021242    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:39.021722 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:39.021314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:39.033763 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:39.033741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:39.425541 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:39.425511    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"]
Sep 15 21:33:39.528239 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:39.528206    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:33:54.445129 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.445071    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn" podUID="19362e19-23c2-4e4b-b7bf-a0d439bb609d" containerName="main" containerID="cri-o://bfbbfc85bd42e419ea41197b525f7bad1141b7c018fb4e5c8d6cbf8f54fff16e" gracePeriod=45
Sep 15 21:33:54.606914 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.606880    2563 generic.go:356] "Generic (PLEG): container finished" podID="19362e19-23c2-4e4b-b7bf-a0d439bb609d" containerID="bfbbfc85bd42e419ea41197b525f7bad1141b7c018fb4e5c8d6cbf8f54fff16e" exitCode=0
Sep 15 21:33:54.607439 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.606921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn" event={"ID":"19362e19-23c2-4e4b-b7bf-a0d439bb609d","Type":"ContainerDied","Data":"bfbbfc85bd42e419ea41197b525f7bad1141b7c018fb4e5c8d6cbf8f54fff16e"}
Sep 15 21:33:54.692097 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.692075    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:33:54.796414 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.796308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6rmn\" (UniqueName: \"kubernetes.io/projected/19362e19-23c2-4e4b-b7bf-a0d439bb609d-kube-api-access-s6rmn\") pod \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") "
Sep 15 21:33:54.796414 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.796369    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-model-cache\") pod \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") "
Sep 15 21:33:54.796414 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.796406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tls-certs\") pod \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") "
Sep 15 21:33:54.796706 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.796558    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-home\") pod \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") "
Sep 15 21:33:54.796706 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.796609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tmp-dir\") pod \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") "
Sep 15 21:33:54.796706 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.796644    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-dshm\") pod \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\" (UID: \"19362e19-23c2-4e4b-b7bf-a0d439bb609d\") "
Sep 15 21:33:54.796706 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.796665    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-model-cache" (OuterVolumeSpecName: "model-cache") pod "19362e19-23c2-4e4b-b7bf-a0d439bb609d" (UID: "19362e19-23c2-4e4b-b7bf-a0d439bb609d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:33:54.796881 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.796817    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-home" (OuterVolumeSpecName: "home") pod "19362e19-23c2-4e4b-b7bf-a0d439bb609d" (UID: "19362e19-23c2-4e4b-b7bf-a0d439bb609d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:33:54.796881 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.796849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19362e19-23c2-4e4b-b7bf-a0d439bb609d" (UID: "19362e19-23c2-4e4b-b7bf-a0d439bb609d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:33:54.797120 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.797103    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:54.797164 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.797122    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:54.797164 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.797134    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:54.798601 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.798578    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-dshm" (OuterVolumeSpecName: "dshm") pod "19362e19-23c2-4e4b-b7bf-a0d439bb609d" (UID: "19362e19-23c2-4e4b-b7bf-a0d439bb609d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:33:54.798822 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.798793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19362e19-23c2-4e4b-b7bf-a0d439bb609d" (UID: "19362e19-23c2-4e4b-b7bf-a0d439bb609d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:33:54.798822 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.798814    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19362e19-23c2-4e4b-b7bf-a0d439bb609d-kube-api-access-s6rmn" (OuterVolumeSpecName: "kube-api-access-s6rmn") pod "19362e19-23c2-4e4b-b7bf-a0d439bb609d" (UID: "19362e19-23c2-4e4b-b7bf-a0d439bb609d"). InnerVolumeSpecName "kube-api-access-s6rmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:33:54.898185 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.898163    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6rmn\" (UniqueName: \"kubernetes.io/projected/19362e19-23c2-4e4b-b7bf-a0d439bb609d-kube-api-access-s6rmn\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:54.898185 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.898182    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19362e19-23c2-4e4b-b7bf-a0d439bb609d-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:54.898319 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:54.898192    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19362e19-23c2-4e4b-b7bf-a0d439bb609d-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:33:55.614777 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:55.614753    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"
Sep 15 21:33:55.615182 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:55.614746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn" event={"ID":"19362e19-23c2-4e4b-b7bf-a0d439bb609d","Type":"ContainerDied","Data":"9479ced6693d93bee7569505863fafc19f59ff8d7c897491bd80bfcd17764508"}
Sep 15 21:33:55.615182 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:55.614876    2563 scope.go:117] "RemoveContainer" containerID="bfbbfc85bd42e419ea41197b525f7bad1141b7c018fb4e5c8d6cbf8f54fff16e"
Sep 15 21:33:55.637049 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:55.637028    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"]
Sep 15 21:33:55.641442 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:55.641422    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d1980a0d/e2e-lifecycle-model-d1980a0d-kserve-7d9467c559-s8ddn"]
Sep 15 21:33:56.525909 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:33:56.525877    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19362e19-23c2-4e4b-b7bf-a0d439bb609d" path="/var/lib/kubelet/pods/19362e19-23c2-4e4b-b7bf-a0d439bb609d/volumes"
Sep 15 21:34:26.821376 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.821328    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"]
Sep 15 21:34:26.822365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.821819    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19362e19-23c2-4e4b-b7bf-a0d439bb609d" containerName="main"
Sep 15 21:34:26.822365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.821832    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="19362e19-23c2-4e4b-b7bf-a0d439bb609d" containerName="main"
Sep 15 21:34:26.822365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.821889    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="19362e19-23c2-4e4b-b7bf-a0d439bb609d" containerName="main"
Sep 15 21:34:26.826522 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.826496    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:26.828890 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.828864    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"kube-root-ca.crt\""
Sep 15 21:34:26.829602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.829578    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"openshift-service-ca.crt\""
Sep 15 21:34:26.830159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.829827    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"auth-test-model-745dff-kserve-self-signed-certs\""
Sep 15 21:34:26.830159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.830060    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"default-dockercfg-clz6r\""
Sep 15 21:34:26.834321 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.834301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"]
Sep 15 21:34:26.898723 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.898684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cbqm\" (UniqueName: \"kubernetes.io/projected/ffa96195-e8d7-42a5-bac8-ba1502d71b49-kube-api-access-9cbqm\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:26.898903 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.898745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-dshm\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:26.898903 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.898779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tls-certs\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:26.898903 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.898816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-model-cache\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:26.899029 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.898933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tmp-dir\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:26.899029 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:26.898985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-home\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.000507 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.000467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-dshm\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.000697 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.000520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tls-certs\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.000697 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.000568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-model-cache\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.000697 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.000616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tmp-dir\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.000697 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.000646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-home\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.000996 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.000770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cbqm\" (UniqueName: \"kubernetes.io/projected/ffa96195-e8d7-42a5-bac8-ba1502d71b49-kube-api-access-9cbqm\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.001057 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.001005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-model-cache\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.001128 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.001072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tmp-dir\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.001308 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.001282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-home\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.003260 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.003236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-dshm\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.003905 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.003884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tls-certs\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.008572 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.008547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cbqm\" (UniqueName: \"kubernetes.io/projected/ffa96195-e8d7-42a5-bac8-ba1502d71b49-kube-api-access-9cbqm\") pod \"auth-test-model-745dff-kserve-55666669ff-57f5k\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") " pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.139597 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.139558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:27.472975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.472945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"]
Sep 15 21:34:27.473861 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:34:27.473823    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffa96195_e8d7_42a5_bac8_ba1502d71b49.slice/crio-c0861df4fdcb35a8f508e40385503602781ed05c7260e66ce1e81202da29ec4b WatchSource:0}: Error finding container c0861df4fdcb35a8f508e40385503602781ed05c7260e66ce1e81202da29ec4b: Status 404 returned error can't find the container with id c0861df4fdcb35a8f508e40385503602781ed05c7260e66ce1e81202da29ec4b
Sep 15 21:34:27.809591 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.809494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k" event={"ID":"ffa96195-e8d7-42a5-bac8-ba1502d71b49","Type":"ContainerStarted","Data":"59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00"}
Sep 15 21:34:27.809591 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.809550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k" event={"ID":"ffa96195-e8d7-42a5-bac8-ba1502d71b49","Type":"ContainerStarted","Data":"c0861df4fdcb35a8f508e40385503602781ed05c7260e66ce1e81202da29ec4b"}
Sep 15 21:34:27.826194 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:27.826144    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k" podStartSLOduration=1.826128929 podStartE2EDuration="1.826128929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:27.82541111 +0000 UTC m=+1457.788468491" watchObservedRunningTime="2026-09-15 21:34:27.826128929 +0000 UTC m=+1457.789186310"
Sep 15 21:34:37.139782 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:37.139748    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:37.140211 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:37.139825    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:37.152101 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:37.152075    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:34:37.881608 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:34:37.881577    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:35:26.479030 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.478990    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"]
Sep 15 21:35:26.482747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.482724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.485303 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.485277    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d8f9d1-serving-cert\""
Sep 15 21:35:26.499165 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.499139    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"]
Sep 15 21:35:26.623636 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.623605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rgk\" (UniqueName: \"kubernetes.io/projected/4e073166-6091-4604-a4a6-df44b0a509e5-kube-api-access-h4rgk\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.623802 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.623783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-metrics-certs\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.623864 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.623816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-maas-api-tls\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.623908 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.623862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e073166-6091-4604-a4a6-df44b0a509e5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.724933 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.724907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-metrics-certs\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.725061 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.724949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-maas-api-tls\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.725061 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.724986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e073166-6091-4604-a4a6-df44b0a509e5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.725061 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.725005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rgk\" (UniqueName: \"kubernetes.io/projected/4e073166-6091-4604-a4a6-df44b0a509e5-kube-api-access-h4rgk\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.725598 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.725583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e073166-6091-4604-a4a6-df44b0a509e5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.727192 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.727165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-maas-api-tls\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.727192 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.727165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-metrics-certs\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.733322 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.733279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rgk\" (UniqueName: \"kubernetes.io/projected/4e073166-6091-4604-a4a6-df44b0a509e5-kube-api-access-h4rgk\") pod \"maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:26.793964 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:26.793943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:27.071106 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:27.071081    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"]
Sep 15 21:35:27.072007 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:35:27.071978    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e073166_6091_4604_a4a6_df44b0a509e5.slice/crio-edf4521c9ba58aefbf1e672308de474cf48cc504e009c550b6c52844c8175379 WatchSource:0}: Error finding container edf4521c9ba58aefbf1e672308de474cf48cc504e009c550b6c52844c8175379: Status 404 returned error can't find the container with id edf4521c9ba58aefbf1e672308de474cf48cc504e009c550b6c52844c8175379
Sep 15 21:35:27.073888 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:27.073862    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:35:27.176137 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:27.176107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn" event={"ID":"4e073166-6091-4604-a4a6-df44b0a509e5","Type":"ContainerStarted","Data":"edf4521c9ba58aefbf1e672308de474cf48cc504e009c550b6c52844c8175379"}
Sep 15 21:35:28.184128 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:28.184092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn" event={"ID":"4e073166-6091-4604-a4a6-df44b0a509e5","Type":"ContainerStarted","Data":"cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52"}
Sep 15 21:35:28.184506 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:28.184267    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:28.204673 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:28.204626    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn" podStartSLOduration=1.7733621290000001 podStartE2EDuration="2.204609888s" totalImagesPullingTime="431.247759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:28.203257915 +0000 UTC m=+1518.166315295" watchObservedRunningTime="2026-09-15 21:35:28.204609888 +0000 UTC m=+1518.167667270"
Sep 15 21:35:34.196324 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:34.196247    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:35:49.330935 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.330893    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"]
Sep 15 21:35:49.334733 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.334712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.338603 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.338583    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-318c22\"/\"embedding-w0-318c22-kserve-self-signed-certs\""
Sep 15 21:35:49.359719 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.359699    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"]
Sep 15 21:35:49.520121 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.520088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-tmp-dir\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.520257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.520154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2fef8e-d640-45b5-a100-a3d36cec5083-tls-certs\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.520257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.520226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-dshm\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.520384 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.520277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-model-cache\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.520384 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.520339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-home\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.520484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.520400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffh8d\" (UniqueName: \"kubernetes.io/projected/3e2fef8e-d640-45b5-a100-a3d36cec5083-kube-api-access-ffh8d\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.620917 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.620888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-tmp-dir\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.621033 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.620939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2fef8e-d640-45b5-a100-a3d36cec5083-tls-certs\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.621033 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.620971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-dshm\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.621033 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.621006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-model-cache\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.621168 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.621045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-home\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.621239 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.621196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffh8d\" (UniqueName: \"kubernetes.io/projected/3e2fef8e-d640-45b5-a100-a3d36cec5083-kube-api-access-ffh8d\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.621335 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.621316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-tmp-dir\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.621436 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.621416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-model-cache\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.621503 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.621459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-home\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.623168 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.623152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-dshm\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.623481 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.623465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2fef8e-d640-45b5-a100-a3d36cec5083-tls-certs\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.629618 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.629595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffh8d\" (UniqueName: \"kubernetes.io/projected/3e2fef8e-d640-45b5-a100-a3d36cec5083-kube-api-access-ffh8d\") pod \"embedding-w0-318c22-kserve-65c887f9f-9hnqx\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") " pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.645229 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.645213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:49.920398 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:49.920368    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"]
Sep 15 21:35:49.920718 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:35:49.920698    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e2fef8e_d640_45b5_a100_a3d36cec5083.slice/crio-50ded1668e319ea9fbd1459e3527e06e380e01d4894845cd4d89b83ba8a10112 WatchSource:0}: Error finding container 50ded1668e319ea9fbd1459e3527e06e380e01d4894845cd4d89b83ba8a10112: Status 404 returned error can't find the container with id 50ded1668e319ea9fbd1459e3527e06e380e01d4894845cd4d89b83ba8a10112
Sep 15 21:35:50.315959 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:50.315881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx" event={"ID":"3e2fef8e-d640-45b5-a100-a3d36cec5083","Type":"ContainerStarted","Data":"81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67"}
Sep 15 21:35:50.315959 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:50.315922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx" event={"ID":"3e2fef8e-d640-45b5-a100-a3d36cec5083","Type":"ContainerStarted","Data":"50ded1668e319ea9fbd1459e3527e06e380e01d4894845cd4d89b83ba8a10112"}
Sep 15 21:35:50.338070 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:50.338024    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx" podStartSLOduration=1.338008125 podStartE2EDuration="1.338008125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:50.336233054 +0000 UTC m=+1540.299290435" watchObservedRunningTime="2026-09-15 21:35:50.338008125 +0000 UTC m=+1540.301065543"
Sep 15 21:35:59.646257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:59.646226    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:59.646257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:59.646263    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:35:59.658379 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:35:59.658335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:36:00.388248 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:00.388203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:36:01.829091 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:01.829062    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"]
Sep 15 21:36:01.833080 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:01.833059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:01.835578 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:01.835489    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-119834-serving-cert\""
Sep 15 21:36:01.843034 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:01.843013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"]
Sep 15 21:36:01.931698 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:01.931671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-metrics-certs\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:01.931823 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:01.931703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn6jr\" (UniqueName: \"kubernetes.io/projected/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-kube-api-access-pn6jr\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:01.931823 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:01.931722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-openshift-service-ca\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:01.931962 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:01.931938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-maas-api-tls\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:02.033155 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:02.033129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-maas-api-tls\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:02.033265 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:02.033173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-metrics-certs\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:02.033310 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:02.033289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn6jr\" (UniqueName: \"kubernetes.io/projected/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-kube-api-access-pn6jr\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:02.033387 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:02.033326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-openshift-service-ca\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:02.033922 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:02.033902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-openshift-service-ca\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:02.035464 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:02.035437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-metrics-certs\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:02.035578 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:02.035563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-maas-api-tls\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:02.042498 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:02.042481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn6jr\" (UniqueName: \"kubernetes.io/projected/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-kube-api-access-pn6jr\") pod \"maas-api-e2e-api-b-119834-56f6446477-jmghb\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:02.147236 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:02.147211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:02.428186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:02.428154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"]
Sep 15 21:36:02.431915 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:36:02.431882    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f87d01_75d9_4b23_82e5_e6750f6fefc9.slice/crio-1364ea4e9761d0e35d500c4e4f03bca8d5f517eb39b1576992898e15f1f066a5 WatchSource:0}: Error finding container 1364ea4e9761d0e35d500c4e4f03bca8d5f517eb39b1576992898e15f1f066a5: Status 404 returned error can't find the container with id 1364ea4e9761d0e35d500c4e4f03bca8d5f517eb39b1576992898e15f1f066a5
Sep 15 21:36:03.398415 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:03.398377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb" event={"ID":"e1f87d01-75d9-4b23-82e5-e6750f6fefc9","Type":"ContainerStarted","Data":"5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1"}
Sep 15 21:36:03.398787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:03.398435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb" event={"ID":"e1f87d01-75d9-4b23-82e5-e6750f6fefc9","Type":"ContainerStarted","Data":"1364ea4e9761d0e35d500c4e4f03bca8d5f517eb39b1576992898e15f1f066a5"}
Sep 15 21:36:03.398787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:03.398455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:03.438527 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:03.438485    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb" podStartSLOduration=1.833153303 podStartE2EDuration="2.438471326s" totalImagesPullingTime="605.318023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:03.436895461 +0000 UTC m=+1553.399952842" watchObservedRunningTime="2026-09-15 21:36:03.438471326 +0000 UTC m=+1553.401528706"
Sep 15 21:36:09.409934 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:09.409903    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:36:48.989079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:48.989048    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"]
Sep 15 21:36:48.989613 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:48.989330    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn" podUID="4e073166-6091-4604-a4a6-df44b0a509e5" containerName="maas-api" containerID="cri-o://cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52" gracePeriod=30
Sep 15 21:36:49.237649 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.237625    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:36:49.350214 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.350145    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4rgk\" (UniqueName: \"kubernetes.io/projected/4e073166-6091-4604-a4a6-df44b0a509e5-kube-api-access-h4rgk\") pod \"4e073166-6091-4604-a4a6-df44b0a509e5\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") "
Sep 15 21:36:49.350214 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.350187    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e073166-6091-4604-a4a6-df44b0a509e5-openshift-service-ca\") pod \"4e073166-6091-4604-a4a6-df44b0a509e5\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") "
Sep 15 21:36:49.350456 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.350374    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-maas-api-tls\") pod \"4e073166-6091-4604-a4a6-df44b0a509e5\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") "
Sep 15 21:36:49.350512 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.350457    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-metrics-certs\") pod \"4e073166-6091-4604-a4a6-df44b0a509e5\" (UID: \"4e073166-6091-4604-a4a6-df44b0a509e5\") "
Sep 15 21:36:49.350679 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.350654    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e073166-6091-4604-a4a6-df44b0a509e5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e073166-6091-4604-a4a6-df44b0a509e5" (UID: "4e073166-6091-4604-a4a6-df44b0a509e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:36:49.351200 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.351179    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e073166-6091-4604-a4a6-df44b0a509e5-openshift-service-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:36:49.352864 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.352831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e073166-6091-4604-a4a6-df44b0a509e5-kube-api-access-h4rgk" (OuterVolumeSpecName: "kube-api-access-h4rgk") pod "4e073166-6091-4604-a4a6-df44b0a509e5" (UID: "4e073166-6091-4604-a4a6-df44b0a509e5"). InnerVolumeSpecName "kube-api-access-h4rgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:49.352987 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.352955    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e073166-6091-4604-a4a6-df44b0a509e5" (UID: "4e073166-6091-4604-a4a6-df44b0a509e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:36:49.353202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.353155    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4e073166-6091-4604-a4a6-df44b0a509e5" (UID: "4e073166-6091-4604-a4a6-df44b0a509e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:36:49.452115 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.452087    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-metrics-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:36:49.452115 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.452113    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rgk\" (UniqueName: \"kubernetes.io/projected/4e073166-6091-4604-a4a6-df44b0a509e5-kube-api-access-h4rgk\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:36:49.452264 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.452122    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e073166-6091-4604-a4a6-df44b0a509e5-maas-api-tls\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:36:49.684947 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.684911    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e073166-6091-4604-a4a6-df44b0a509e5" containerID="cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52" exitCode=0
Sep 15 21:36:49.685111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.684954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn" event={"ID":"4e073166-6091-4604-a4a6-df44b0a509e5","Type":"ContainerDied","Data":"cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52"}
Sep 15 21:36:49.685111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.685001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn" event={"ID":"4e073166-6091-4604-a4a6-df44b0a509e5","Type":"ContainerDied","Data":"edf4521c9ba58aefbf1e672308de474cf48cc504e009c550b6c52844c8175379"}
Sep 15 21:36:49.685111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.685021    2563 scope.go:117] "RemoveContainer" containerID="cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52"
Sep 15 21:36:49.685111 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.684969    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"
Sep 15 21:36:49.695636 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.695621    2563 scope.go:117] "RemoveContainer" containerID="cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52"
Sep 15 21:36:49.695884 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:36:49.695866    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52\": container with ID starting with cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52 not found: ID does not exist" containerID="cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52"
Sep 15 21:36:49.695943 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.695891    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52"} err="failed to get container status \"cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52\": rpc error: code = NotFound desc = could not find container \"cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52\": container with ID starting with cd00fb01c1387464f1f7371012e2ead79a36f9a3aa9a59efe3c01288d4127d52 not found: ID does not exist"
Sep 15 21:36:49.708358 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.708324    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"]
Sep 15 21:36:49.712489 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:49.712468    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn"]
Sep 15 21:36:50.526653 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:50.526622    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e073166-6091-4604-a4a6-df44b0a509e5" path="/var/lib/kubelet/pods/4e073166-6091-4604-a4a6-df44b0a509e5/volumes"
Sep 15 21:36:52.192043 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:36:52.192007    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d8f9d1-fd66b6c74-c2fxn" podUID="4e073166-6091-4604-a4a6-df44b0a509e5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.86:8443/health\": context deadline exceeded"
Sep 15 21:37:12.831037 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:12.831002    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"]
Sep 15 21:37:12.831450 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:12.831256    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb" podUID="e1f87d01-75d9-4b23-82e5-e6750f6fefc9" containerName="maas-api" containerID="cri-o://5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1" gracePeriod=30
Sep 15 21:37:13.079538 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.079515    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:37:13.162243 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.162208    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-metrics-certs\") pod \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") "
Sep 15 21:37:13.162449 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.162383    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-openshift-service-ca\") pod \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") "
Sep 15 21:37:13.162449 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.162434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn6jr\" (UniqueName: \"kubernetes.io/projected/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-kube-api-access-pn6jr\") pod \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") "
Sep 15 21:37:13.162557 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.162517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-maas-api-tls\") pod \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\" (UID: \"e1f87d01-75d9-4b23-82e5-e6750f6fefc9\") "
Sep 15 21:37:13.162779 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.162724    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1f87d01-75d9-4b23-82e5-e6750f6fefc9" (UID: "e1f87d01-75d9-4b23-82e5-e6750f6fefc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:13.162941 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.162920    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-openshift-service-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:37:13.164458 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.164433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e1f87d01-75d9-4b23-82e5-e6750f6fefc9" (UID: "e1f87d01-75d9-4b23-82e5-e6750f6fefc9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:37:13.164545 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.164510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-kube-api-access-pn6jr" (OuterVolumeSpecName: "kube-api-access-pn6jr") pod "e1f87d01-75d9-4b23-82e5-e6750f6fefc9" (UID: "e1f87d01-75d9-4b23-82e5-e6750f6fefc9"). InnerVolumeSpecName "kube-api-access-pn6jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:13.164910 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.164891    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1f87d01-75d9-4b23-82e5-e6750f6fefc9" (UID: "e1f87d01-75d9-4b23-82e5-e6750f6fefc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:37:13.264403 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.264378    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn6jr\" (UniqueName: \"kubernetes.io/projected/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-kube-api-access-pn6jr\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:37:13.264403 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.264400    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-maas-api-tls\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:37:13.264548 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.264410    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f87d01-75d9-4b23-82e5-e6750f6fefc9-metrics-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:37:13.849819 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.849788    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1f87d01-75d9-4b23-82e5-e6750f6fefc9" containerID="5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1" exitCode=0
Sep 15 21:37:13.850233 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.849832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb" event={"ID":"e1f87d01-75d9-4b23-82e5-e6750f6fefc9","Type":"ContainerDied","Data":"5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1"}
Sep 15 21:37:13.850233 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.849838    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"
Sep 15 21:37:13.850233 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.849875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb" event={"ID":"e1f87d01-75d9-4b23-82e5-e6750f6fefc9","Type":"ContainerDied","Data":"1364ea4e9761d0e35d500c4e4f03bca8d5f517eb39b1576992898e15f1f066a5"}
Sep 15 21:37:13.850233 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.849897    2563 scope.go:117] "RemoveContainer" containerID="5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1"
Sep 15 21:37:13.860160 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.860143    2563 scope.go:117] "RemoveContainer" containerID="5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1"
Sep 15 21:37:13.860471 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:37:13.860455    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1\": container with ID starting with 5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1 not found: ID does not exist" containerID="5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1"
Sep 15 21:37:13.860549 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.860482    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1"} err="failed to get container status \"5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1\": rpc error: code = NotFound desc = could not find container \"5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1\": container with ID starting with 5b17f0b7f2d4a1fa8b48ba02c0a777163e7173654c51e888fa6440e0d6264ca1 not found: ID does not exist"
Sep 15 21:37:13.873162 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.873141    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"]
Sep 15 21:37:13.879586 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:13.879567    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-119834-56f6446477-jmghb"]
Sep 15 21:37:14.527509 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:14.527480    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f87d01-75d9-4b23-82e5-e6750f6fefc9" path="/var/lib/kubelet/pods/e1f87d01-75d9-4b23-82e5-e6750f6fefc9/volumes"
Sep 15 21:37:31.031595 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:31.031558    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542"]
Sep 15 21:37:31.038483 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:31.038456    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbcc8f-29825130-dl542"]
Sep 15 21:37:32.526708 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:32.526680    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc4c4515-f093-4e89-9c5c-2d1559c32f24" path="/var/lib/kubelet/pods/fc4c4515-f093-4e89-9c5c-2d1559c32f24/volumes"
Sep 15 21:37:45.015775 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:45.015729    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"]
Sep 15 21:37:45.021671 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:45.021646    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"]
Sep 15 21:37:45.030358 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:45.030320    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"]
Sep 15 21:37:45.040713 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:45.040688    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"]
Sep 15 21:37:45.045895 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:45.045868    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"]
Sep 15 21:37:57.987672 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:57.987637    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"]
Sep 15 21:37:57.988128 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:57.987877    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp" podUID="af87cba1-663c-432f-adf7-bf25a39bf502" containerName="payload-pre-processing" containerID="cri-o://d5534b205492bc73c6c1195dd1ddce829c70e42e5f59eeccec94f0e43cabb52c" gracePeriod=30
Sep 15 21:37:58.127498 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:58.127460    2563 generic.go:356] "Generic (PLEG): container finished" podID="af87cba1-663c-432f-adf7-bf25a39bf502" containerID="d5534b205492bc73c6c1195dd1ddce829c70e42e5f59eeccec94f0e43cabb52c" exitCode=0
Sep 15 21:37:58.127661 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:58.127528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp" event={"ID":"af87cba1-663c-432f-adf7-bf25a39bf502","Type":"ContainerDied","Data":"d5534b205492bc73c6c1195dd1ddce829c70e42e5f59eeccec94f0e43cabb52c"}
Sep 15 21:37:58.248595 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:58.248521    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:37:58.410788 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:58.410755    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af87cba1-663c-432f-adf7-bf25a39bf502-plugins-config-volume\") pod \"af87cba1-663c-432f-adf7-bf25a39bf502\" (UID: \"af87cba1-663c-432f-adf7-bf25a39bf502\") "
Sep 15 21:37:58.410964 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:58.410814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plfsz\" (UniqueName: \"kubernetes.io/projected/af87cba1-663c-432f-adf7-bf25a39bf502-kube-api-access-plfsz\") pod \"af87cba1-663c-432f-adf7-bf25a39bf502\" (UID: \"af87cba1-663c-432f-adf7-bf25a39bf502\") "
Sep 15 21:37:58.411116 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:58.411092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af87cba1-663c-432f-adf7-bf25a39bf502-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af87cba1-663c-432f-adf7-bf25a39bf502" (UID: "af87cba1-663c-432f-adf7-bf25a39bf502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:58.412869 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:58.412841    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af87cba1-663c-432f-adf7-bf25a39bf502-kube-api-access-plfsz" (OuterVolumeSpecName: "kube-api-access-plfsz") pod "af87cba1-663c-432f-adf7-bf25a39bf502" (UID: "af87cba1-663c-432f-adf7-bf25a39bf502"). InnerVolumeSpecName "kube-api-access-plfsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:58.512330 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:58.512243    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af87cba1-663c-432f-adf7-bf25a39bf502-plugins-config-volume\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:37:58.512330 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:58.512271    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plfsz\" (UniqueName: \"kubernetes.io/projected/af87cba1-663c-432f-adf7-bf25a39bf502-kube-api-access-plfsz\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:37:59.136264 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:59.136236    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"
Sep 15 21:37:59.136710 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:59.136276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp" event={"ID":"af87cba1-663c-432f-adf7-bf25a39bf502","Type":"ContainerDied","Data":"2242aa094fc80c03fd67ea10de6074cc40728ea14df6528a3aae94f0dc6b047b"}
Sep 15 21:37:59.136710 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:59.136335    2563 scope.go:117] "RemoveContainer" containerID="d5534b205492bc73c6c1195dd1ddce829c70e42e5f59eeccec94f0e43cabb52c"
Sep 15 21:37:59.154554 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:59.154525    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"]
Sep 15 21:37:59.157185 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:37:59.157165    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-318c22-6bdcbc9474-r9vjp"]
Sep 15 21:38:00.037398 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.037339    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv" podUID="b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" containerName="main" containerID="cri-o://cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db" gracePeriod=45
Sep 15 21:38:00.042228 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.042197    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n" podUID="3f170c4b-dff1-44fc-8081-4a7785446f59" containerName="main" containerID="cri-o://57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871" gracePeriod=45
Sep 15 21:38:00.050464 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.050430    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx" podUID="3e2fef8e-d640-45b5-a100-a3d36cec5083" containerName="main" containerID="cri-o://81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67" gracePeriod=45
Sep 15 21:38:00.069743 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.069709    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7" podUID="d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" containerName="main" containerID="cri-o://1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0" gracePeriod=45
Sep 15 21:38:00.069853 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.069720    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26" podUID="ae15f1b9-db5b-4f74-b8ee-61cf58147429" containerName="main" containerID="cri-o://29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90" gracePeriod=45
Sep 15 21:38:00.368849 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.368822    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:38:00.391928 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.391712    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:38:00.403206 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.403184    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:38:00.406014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.405993    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:38:00.409835 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.409817    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:38:00.526578 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.526549    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af87cba1-663c-432f-adf7-bf25a39bf502" path="/var/lib/kubelet/pods/af87cba1-663c-432f-adf7-bf25a39bf502/volumes"
Sep 15 21:38:00.533339 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533317    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-tmp-dir\") pod \"3f170c4b-dff1-44fc-8081-4a7785446f59\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") "
Sep 15 21:38:00.533477 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533360    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tls-certs\") pod \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") "
Sep 15 21:38:00.533477 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-home\") pod \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") "
Sep 15 21:38:00.533477 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpcv8\" (UniqueName: \"kubernetes.io/projected/3f170c4b-dff1-44fc-8081-4a7785446f59-kube-api-access-zpcv8\") pod \"3f170c4b-dff1-44fc-8081-4a7785446f59\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") "
Sep 15 21:38:00.533477 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-model-cache\") pod \"3e2fef8e-d640-45b5-a100-a3d36cec5083\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") "
Sep 15 21:38:00.533477 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tmp-dir\") pod \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") "
Sep 15 21:38:00.533477 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533460    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6s58\" (UniqueName: \"kubernetes.io/projected/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-kube-api-access-z6s58\") pod \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") "
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533484    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-dshm\") pod \"3e2fef8e-d640-45b5-a100-a3d36cec5083\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") "
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533515    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-home\") pod \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") "
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533534    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f170c4b-dff1-44fc-8081-4a7785446f59" (UID: "3f170c4b-dff1-44fc-8081-4a7785446f59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533559    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-home\") pod \"3f170c4b-dff1-44fc-8081-4a7785446f59\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") "
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533594    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f170c4b-dff1-44fc-8081-4a7785446f59-tls-certs\") pod \"3f170c4b-dff1-44fc-8081-4a7785446f59\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") "
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-model-cache\") pod \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") "
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-dshm\") pod \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") "
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533663    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e2fef8e-d640-45b5-a100-a3d36cec5083" (UID: "3e2fef8e-d640-45b5-a100-a3d36cec5083"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533675    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-dshm\") pod \"3f170c4b-dff1-44fc-8081-4a7785446f59\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") "
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533693    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-home" (OuterVolumeSpecName: "home") pod "ae15f1b9-db5b-4f74-b8ee-61cf58147429" (UID: "ae15f1b9-db5b-4f74-b8ee-61cf58147429"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-home\") pod \"3e2fef8e-d640-45b5-a100-a3d36cec5083\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") "
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533758    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-model-cache\") pod \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") "
Sep 15 21:38:00.533804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhbvg\" (UniqueName: \"kubernetes.io/projected/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-kube-api-access-mhbvg\") pod \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533869    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tls-certs\") pod \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-home" (OuterVolumeSpecName: "home") pod "3e2fef8e-d640-45b5-a100-a3d36cec5083" (UID: "3e2fef8e-d640-45b5-a100-a3d36cec5083"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tls-certs\") pod \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533928    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-home\") pod \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533958    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-dshm\") pod \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533992    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-model-cache\") pod \"3f170c4b-dff1-44fc-8081-4a7785446f59\" (UID: \"3f170c4b-dff1-44fc-8081-4a7785446f59\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.533991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae15f1b9-db5b-4f74-b8ee-61cf58147429" (UID: "ae15f1b9-db5b-4f74-b8ee-61cf58147429"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534018    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-dshm\") pod \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534044    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-tmp-dir\") pod \"3e2fef8e-d640-45b5-a100-a3d36cec5083\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tmp-dir\") pod \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\" (UID: \"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534096    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" (UID: "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534115    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2fef8e-d640-45b5-a100-a3d36cec5083-tls-certs\") pod \"3e2fef8e-d640-45b5-a100-a3d36cec5083\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534152    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crmd9\" (UniqueName: \"kubernetes.io/projected/ae15f1b9-db5b-4f74-b8ee-61cf58147429-kube-api-access-crmd9\") pod \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\" (UID: \"ae15f1b9-db5b-4f74-b8ee-61cf58147429\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534184    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-model-cache\") pod \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534214    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffh8d\" (UniqueName: \"kubernetes.io/projected/3e2fef8e-d640-45b5-a100-a3d36cec5083-kube-api-access-ffh8d\") pod \"3e2fef8e-d640-45b5-a100-a3d36cec5083\" (UID: \"3e2fef8e-d640-45b5-a100-a3d36cec5083\") "
Sep 15 21:38:00.534471 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534291    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tmp-dir\") pod \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\" (UID: \"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa\") "
Sep 15 21:38:00.535250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534689    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.535250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534707    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.535250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534721    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.535250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534734    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.535250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534746    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.535250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534758    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.535250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.534960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" (UID: "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.535250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.535137    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-home" (OuterVolumeSpecName: "home") pod "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" (UID: "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.536189 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.536162    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f170c4b-dff1-44fc-8081-4a7785446f59-kube-api-access-zpcv8" (OuterVolumeSpecName: "kube-api-access-zpcv8") pod "3f170c4b-dff1-44fc-8081-4a7785446f59" (UID: "3f170c4b-dff1-44fc-8081-4a7785446f59"). InnerVolumeSpecName "kube-api-access-zpcv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:00.536414 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.536387    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" (UID: "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.536414 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.536406    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-dshm" (OuterVolumeSpecName: "dshm") pod "3e2fef8e-d640-45b5-a100-a3d36cec5083" (UID: "3e2fef8e-d640-45b5-a100-a3d36cec5083"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.536590 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.536413    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-dshm" (OuterVolumeSpecName: "dshm") pod "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" (UID: "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.536590 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.536429    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f170c4b-dff1-44fc-8081-4a7785446f59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f170c4b-dff1-44fc-8081-4a7785446f59" (UID: "3f170c4b-dff1-44fc-8081-4a7785446f59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:00.536755 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.536597    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-home" (OuterVolumeSpecName: "home") pod "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" (UID: "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.536755 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.536598    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f170c4b-dff1-44fc-8081-4a7785446f59" (UID: "3f170c4b-dff1-44fc-8081-4a7785446f59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.536755 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.536610    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" (UID: "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:00.536755 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.536723    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-home" (OuterVolumeSpecName: "home") pod "3f170c4b-dff1-44fc-8081-4a7785446f59" (UID: "3f170c4b-dff1-44fc-8081-4a7785446f59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.536755 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.536750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" (UID: "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:00.538675 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.538025    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-kube-api-access-z6s58" (OuterVolumeSpecName: "kube-api-access-z6s58") pod "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" (UID: "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa"). InnerVolumeSpecName "kube-api-access-z6s58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:00.538675 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.538143    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-dshm" (OuterVolumeSpecName: "dshm") pod "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" (UID: "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.538675 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.538366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae15f1b9-db5b-4f74-b8ee-61cf58147429" (UID: "ae15f1b9-db5b-4f74-b8ee-61cf58147429"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.538675 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.538368    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" (UID: "d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.538675 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.538616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e2fef8e-d640-45b5-a100-a3d36cec5083" (UID: "3e2fef8e-d640-45b5-a100-a3d36cec5083"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.538960 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.538894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae15f1b9-db5b-4f74-b8ee-61cf58147429-kube-api-access-crmd9" (OuterVolumeSpecName: "kube-api-access-crmd9") pod "ae15f1b9-db5b-4f74-b8ee-61cf58147429" (UID: "ae15f1b9-db5b-4f74-b8ee-61cf58147429"). InnerVolumeSpecName "kube-api-access-crmd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:00.538960 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.538906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-kube-api-access-mhbvg" (OuterVolumeSpecName: "kube-api-access-mhbvg") pod "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" (UID: "b0c7e985-13f4-44f6-ada0-8aae1c5cedcb"). InnerVolumeSpecName "kube-api-access-mhbvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:00.539033 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.539018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-dshm" (OuterVolumeSpecName: "dshm") pod "ae15f1b9-db5b-4f74-b8ee-61cf58147429" (UID: "ae15f1b9-db5b-4f74-b8ee-61cf58147429"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.539143 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.539125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-dshm" (OuterVolumeSpecName: "dshm") pod "3f170c4b-dff1-44fc-8081-4a7785446f59" (UID: "3f170c4b-dff1-44fc-8081-4a7785446f59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:00.540009 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.539991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae15f1b9-db5b-4f74-b8ee-61cf58147429" (UID: "ae15f1b9-db5b-4f74-b8ee-61cf58147429"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:00.540223 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.540205    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e2fef8e-d640-45b5-a100-a3d36cec5083-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e2fef8e-d640-45b5-a100-a3d36cec5083" (UID: "3e2fef8e-d640-45b5-a100-a3d36cec5083"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:00.540270 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.540205    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2fef8e-d640-45b5-a100-a3d36cec5083-kube-api-access-ffh8d" (OuterVolumeSpecName: "kube-api-access-ffh8d") pod "3e2fef8e-d640-45b5-a100-a3d36cec5083" (UID: "3e2fef8e-d640-45b5-a100-a3d36cec5083"). InnerVolumeSpecName "kube-api-access-ffh8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:00.635900 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635864    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmd9\" (UniqueName: \"kubernetes.io/projected/ae15f1b9-db5b-4f74-b8ee-61cf58147429-kube-api-access-crmd9\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.635900 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635887    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.635900 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635901    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffh8d\" (UniqueName: \"kubernetes.io/projected/3e2fef8e-d640-45b5-a100-a3d36cec5083-kube-api-access-ffh8d\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635913    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635927    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635938    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpcv8\" (UniqueName: \"kubernetes.io/projected/3f170c4b-dff1-44fc-8081-4a7785446f59-kube-api-access-zpcv8\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635950    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6s58\" (UniqueName: \"kubernetes.io/projected/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-kube-api-access-z6s58\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635962    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635972    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635983    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.635993    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f170c4b-dff1-44fc-8081-4a7785446f59-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636004    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636014    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636026    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636038    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbvg\" (UniqueName: \"kubernetes.io/projected/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-kube-api-access-mhbvg\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636049    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae15f1b9-db5b-4f74-b8ee-61cf58147429-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636063    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636074    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636084    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636094    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f170c4b-dff1-44fc-8081-4a7785446f59-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636105    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae15f1b9-db5b-4f74-b8ee-61cf58147429-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636115    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2fef8e-d640-45b5-a100-a3d36cec5083-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636126    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:00.636202 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:00.636137    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2fef8e-d640-45b5-a100-a3d36cec5083-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:01.150944 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.150910    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e2fef8e-d640-45b5-a100-a3d36cec5083" containerID="81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67" exitCode=0
Sep 15 21:38:01.151119 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.150957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"
Sep 15 21:38:01.151119 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.150994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx" event={"ID":"3e2fef8e-d640-45b5-a100-a3d36cec5083","Type":"ContainerDied","Data":"81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67"}
Sep 15 21:38:01.151119 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.151039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx" event={"ID":"3e2fef8e-d640-45b5-a100-a3d36cec5083","Type":"ContainerDied","Data":"50ded1668e319ea9fbd1459e3527e06e380e01d4894845cd4d89b83ba8a10112"}
Sep 15 21:38:01.151119 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.151058    2563 scope.go:117] "RemoveContainer" containerID="81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67"
Sep 15 21:38:01.152532 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.152504    2563 generic.go:356] "Generic (PLEG): container finished" podID="d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" containerID="1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0" exitCode=0
Sep 15 21:38:01.152657 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.152547    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"
Sep 15 21:38:01.152657 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.152593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7" event={"ID":"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa","Type":"ContainerDied","Data":"1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0"}
Sep 15 21:38:01.152657 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.152621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7" event={"ID":"d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa","Type":"ContainerDied","Data":"e99f7ff939229d1e30d07d385a6b98b1e7a814de8b03633167ed714b010dca79"}
Sep 15 21:38:01.153928 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.153906    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae15f1b9-db5b-4f74-b8ee-61cf58147429" containerID="29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90" exitCode=0
Sep 15 21:38:01.154043 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.153942    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"
Sep 15 21:38:01.154043 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.153994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26" event={"ID":"ae15f1b9-db5b-4f74-b8ee-61cf58147429","Type":"ContainerDied","Data":"29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90"}
Sep 15 21:38:01.154317 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.154282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26" event={"ID":"ae15f1b9-db5b-4f74-b8ee-61cf58147429","Type":"ContainerDied","Data":"fd3ba984e124bde8e20874373b0ebc93015a3206785e2470735746b02a63c078"}
Sep 15 21:38:01.155806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.155673    2563 generic.go:356] "Generic (PLEG): container finished" podID="b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" containerID="cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db" exitCode=0
Sep 15 21:38:01.155806 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.155725    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"
Sep 15 21:38:01.155955 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.155772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv" event={"ID":"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb","Type":"ContainerDied","Data":"cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db"}
Sep 15 21:38:01.163643 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.162646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv" event={"ID":"b0c7e985-13f4-44f6-ada0-8aae1c5cedcb","Type":"ContainerDied","Data":"6e961f31fe4f8c157980606ac7c50f8e151d5d6d1e3656d67e1f12f203add370"}
Sep 15 21:38:01.165287 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.165151    2563 scope.go:117] "RemoveContainer" containerID="81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67"
Sep 15 21:38:01.166535 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:38:01.166504    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67\": container with ID starting with 81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67 not found: ID does not exist" containerID="81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67"
Sep 15 21:38:01.166657 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.166545    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67"} err="failed to get container status \"81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67\": rpc error: code = NotFound desc = could not find container \"81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67\": container with ID starting with 81ba2d8dafbceb631b737c902f5605d5bbbd5734813975d0e5fcb16d4c202c67 not found: ID does not exist"
Sep 15 21:38:01.166657 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.166583    2563 scope.go:117] "RemoveContainer" containerID="1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0"
Sep 15 21:38:01.167727 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.167585    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f170c4b-dff1-44fc-8081-4a7785446f59" containerID="57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871" exitCode=0
Sep 15 21:38:01.167727 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.167653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n" event={"ID":"3f170c4b-dff1-44fc-8081-4a7785446f59","Type":"ContainerDied","Data":"57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871"}
Sep 15 21:38:01.167727 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.167679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n" event={"ID":"3f170c4b-dff1-44fc-8081-4a7785446f59","Type":"ContainerDied","Data":"a7ffff45a2c5a8fd63f919dcbcdd7da912c35eac41579182ea46f450151eb75a"}
Sep 15 21:38:01.167912 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.167821    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"
Sep 15 21:38:01.180022 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.179890    2563 scope.go:117] "RemoveContainer" containerID="1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0"
Sep 15 21:38:01.180164 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:38:01.180144    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0\": container with ID starting with 1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0 not found: ID does not exist" containerID="1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0"
Sep 15 21:38:01.180207 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.180174    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0"} err="failed to get container status \"1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0\": rpc error: code = NotFound desc = could not find container \"1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0\": container with ID starting with 1d3735bac2b9bbbe1e1d5a4e2726e8ba1f6f2abde908dcaecef6b603828ab1f0 not found: ID does not exist"
Sep 15 21:38:01.180207 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.180193    2563 scope.go:117] "RemoveContainer" containerID="29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90"
Sep 15 21:38:01.188369 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.188268    2563 scope.go:117] "RemoveContainer" containerID="29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90"
Sep 15 21:38:01.189363 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:38:01.188695    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90\": container with ID starting with 29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90 not found: ID does not exist" containerID="29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90"
Sep 15 21:38:01.189363 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.188723    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90"} err="failed to get container status \"29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90\": rpc error: code = NotFound desc = could not find container \"29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90\": container with ID starting with 29a95888dd593f475a7ff977ec3ad02161726526cfc756b329796e8ce553be90 not found: ID does not exist"
Sep 15 21:38:01.189363 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.188742    2563 scope.go:117] "RemoveContainer" containerID="cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db"
Sep 15 21:38:01.191804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.191782    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"]
Sep 15 21:38:01.196193 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.196167    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/simulator-w0-318c22-kserve-6fb4f8699d-8twf7"]
Sep 15 21:38:01.198247 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.198230    2563 scope.go:117] "RemoveContainer" containerID="cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db"
Sep 15 21:38:01.198600 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:38:01.198583    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db\": container with ID starting with cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db not found: ID does not exist" containerID="cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db"
Sep 15 21:38:01.198654 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.198605    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db"} err="failed to get container status \"cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db\": rpc error: code = NotFound desc = could not find container \"cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db\": container with ID starting with cf5ce0a2f93fb99f75a67dd2e10ff84c2c812ab3d5da88603aa35d66b3f4a0db not found: ID does not exist"
Sep 15 21:38:01.198654 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.198622    2563 scope.go:117] "RemoveContainer" containerID="57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871"
Sep 15 21:38:01.207286 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.207269    2563 scope.go:117] "RemoveContainer" containerID="57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871"
Sep 15 21:38:01.207587 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:38:01.207564    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871\": container with ID starting with 57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871 not found: ID does not exist" containerID="57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871"
Sep 15 21:38:01.207647 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.207596    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871"} err="failed to get container status \"57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871\": rpc error: code = NotFound desc = could not find container \"57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871\": container with ID starting with 57303e067e029d133a33456c3f17865977413a8bec547155272d63584688e871 not found: ID does not exist"
Sep 15 21:38:01.213609 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.213590    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"]
Sep 15 21:38:01.217385 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.217366    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/embedding-w0-318c22-kserve-65c887f9f-9hnqx"]
Sep 15 21:38:01.230016 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.229992    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"]
Sep 15 21:38:01.234312 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.234294    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/premium-w0-318c22-kserve-5cd9d7b74c-lpj26"]
Sep 15 21:38:01.244162 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.244142    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"]
Sep 15 21:38:01.248719 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.248701    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-w0-318c22-kserve-c4cbcf446-2t47n"]
Sep 15 21:38:01.258843 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.258823    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"]
Sep 15 21:38:01.262548 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:01.262529    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv"]
Sep 15 21:38:02.526525 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:02.526492    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e2fef8e-d640-45b5-a100-a3d36cec5083" path="/var/lib/kubelet/pods/3e2fef8e-d640-45b5-a100-a3d36cec5083/volumes"
Sep 15 21:38:02.526912 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:02.526898    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f170c4b-dff1-44fc-8081-4a7785446f59" path="/var/lib/kubelet/pods/3f170c4b-dff1-44fc-8081-4a7785446f59/volumes"
Sep 15 21:38:02.527254 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:02.527242    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae15f1b9-db5b-4f74-b8ee-61cf58147429" path="/var/lib/kubelet/pods/ae15f1b9-db5b-4f74-b8ee-61cf58147429/volumes"
Sep 15 21:38:02.527924 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:02.527907    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" path="/var/lib/kubelet/pods/b0c7e985-13f4-44f6-ada0-8aae1c5cedcb/volumes"
Sep 15 21:38:02.528306 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:02.528293    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" path="/var/lib/kubelet/pods/d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa/volumes"
Sep 15 21:38:05.397785 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:05.397738    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-318c22/distinct-2-w0-318c22-kserve-576b7777f7-gk4jv" podUID="b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 15 21:38:10.784060 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:10.784026    2563 scope.go:117] "RemoveContainer" containerID="749cb283191cfb431c717b3fc3b9ecfee4f03e1ac071de553dc679a30cac8c0b"
Sep 15 21:38:33.893078 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:33.893001    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"]
Sep 15 21:38:48.912154 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:48.912110    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv" podUID="5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" containerName="main" containerID="cri-o://2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b" gracePeriod=45
Sep 15 21:38:49.160656 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.160628    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:38:49.205975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.205868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tmp-dir\") pod \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") "
Sep 15 21:38:49.205975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.205919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp8zl\" (UniqueName: \"kubernetes.io/projected/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-kube-api-access-jp8zl\") pod \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") "
Sep 15 21:38:49.205975 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.205950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-home\") pod \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") "
Sep 15 21:38:49.206265 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.206037    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tls-certs\") pod \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") "
Sep 15 21:38:49.206265 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.206075    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-dshm\") pod \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") "
Sep 15 21:38:49.206265 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.206118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-model-cache\") pod \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\" (UID: \"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d\") "
Sep 15 21:38:49.206265 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.206177    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" (UID: "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.206489 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.206310    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-home" (OuterVolumeSpecName: "home") pod "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" (UID: "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.206489 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.206432    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" (UID: "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.206602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.206522    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.206602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.206548    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.206602 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.206562    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.208394 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.208322    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-dshm" (OuterVolumeSpecName: "dshm") pod "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" (UID: "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.208548 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.208451    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" (UID: "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:49.208837 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.208809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-kube-api-access-jp8zl" (OuterVolumeSpecName: "kube-api-access-jp8zl") pod "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" (UID: "5e32eff9-bb23-4daa-8cc3-f5e3ae09488d"). InnerVolumeSpecName "kube-api-access-jp8zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:49.307803 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.307768    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp8zl\" (UniqueName: \"kubernetes.io/projected/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-kube-api-access-jp8zl\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.307803 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.307800    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.307803 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.307811    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.463014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.462930    2563 generic.go:356] "Generic (PLEG): container finished" podID="5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" containerID="2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b" exitCode=0
Sep 15 21:38:49.463014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.462978    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"
Sep 15 21:38:49.463194 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.463012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv" event={"ID":"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d","Type":"ContainerDied","Data":"2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b"}
Sep 15 21:38:49.463194 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.463059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv" event={"ID":"5e32eff9-bb23-4daa-8cc3-f5e3ae09488d","Type":"ContainerDied","Data":"da333b4fd199dd28faceb2cb0fc1dd5b15ddf2445fe6212d5f67e8782353e639"}
Sep 15 21:38:49.463194 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.463078    2563 scope.go:117] "RemoveContainer" containerID="2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b"
Sep 15 21:38:49.472946 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.472925    2563 scope.go:117] "RemoveContainer" containerID="2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b"
Sep 15 21:38:49.473246 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:38:49.473227    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b\": container with ID starting with 2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b not found: ID does not exist" containerID="2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b"
Sep 15 21:38:49.473357 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.473258    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b"} err="failed to get container status \"2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b\": rpc error: code = NotFound desc = could not find container \"2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b\": container with ID starting with 2dcd07df949ab3478d6386ba64784f9ada50938810a507808b4814d619e7c32b not found: ID does not exist"
Sep 15 21:38:49.484323 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.484290    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"]
Sep 15 21:38:49.488596 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:49.488569    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/simulator-w4-6f66f7-kserve-786c449848-p2qkv"]
Sep 15 21:38:50.528663 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:38:50.528626    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" path="/var/lib/kubelet/pods/5e32eff9-bb23-4daa-8cc3-f5e3ae09488d/volumes"
Sep 15 21:39:05.301245 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:05.301213    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"]
Sep 15 21:39:06.697275 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:06.697242    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"]
Sep 15 21:39:20.320064 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.320011    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb" podUID="4e56b4af-b11d-4774-962c-3f7a9f4523e5" containerName="main" containerID="cri-o://bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9" gracePeriod=45
Sep 15 21:39:20.566509 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.566488    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:39:20.637831 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.637801    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e56b4af-b11d-4774-962c-3f7a9f4523e5" containerID="bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9" exitCode=0
Sep 15 21:39:20.637972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.637843    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"
Sep 15 21:39:20.637972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.637883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb" event={"ID":"4e56b4af-b11d-4774-962c-3f7a9f4523e5","Type":"ContainerDied","Data":"bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9"}
Sep 15 21:39:20.637972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.637925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb" event={"ID":"4e56b4af-b11d-4774-962c-3f7a9f4523e5","Type":"ContainerDied","Data":"9aea52e5876bdd8920a4aa7c5c447882eea242b3378853ddff549ba2f36b2a3b"}
Sep 15 21:39:20.637972 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.637941    2563 scope.go:117] "RemoveContainer" containerID="bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9"
Sep 15 21:39:20.646661 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.646645    2563 scope.go:117] "RemoveContainer" containerID="bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9"
Sep 15 21:39:20.646932 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:39:20.646913    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9\": container with ID starting with bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9 not found: ID does not exist" containerID="bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9"
Sep 15 21:39:20.646982 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.646941    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9"} err="failed to get container status \"bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9\": rpc error: code = NotFound desc = could not find container \"bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9\": container with ID starting with bec850440373bc917fb5f078af0601cab3b0aed0f0fee61ca32f3a124ed012c9 not found: ID does not exist"
Sep 15 21:39:20.686133 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-dshm\") pod \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") "
Sep 15 21:39:20.686238 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686153    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-model-cache\") pod \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") "
Sep 15 21:39:20.686297 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tls-certs\") pod \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") "
Sep 15 21:39:20.686381 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686295    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggrvp\" (UniqueName: \"kubernetes.io/projected/4e56b4af-b11d-4774-962c-3f7a9f4523e5-kube-api-access-ggrvp\") pod \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") "
Sep 15 21:39:20.686381 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-home\") pod \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") "
Sep 15 21:39:20.686495 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686427    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tmp-dir\") pod \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\" (UID: \"4e56b4af-b11d-4774-962c-3f7a9f4523e5\") "
Sep 15 21:39:20.686495 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686470    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e56b4af-b11d-4774-962c-3f7a9f4523e5" (UID: "4e56b4af-b11d-4774-962c-3f7a9f4523e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:39:20.686675 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686655    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-home" (OuterVolumeSpecName: "home") pod "4e56b4af-b11d-4774-962c-3f7a9f4523e5" (UID: "4e56b4af-b11d-4774-962c-3f7a9f4523e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:39:20.686789 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686750    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:20.686851 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686796    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:20.686851 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.686808    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e56b4af-b11d-4774-962c-3f7a9f4523e5" (UID: "4e56b4af-b11d-4774-962c-3f7a9f4523e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:39:20.688282 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.688263    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-dshm" (OuterVolumeSpecName: "dshm") pod "4e56b4af-b11d-4774-962c-3f7a9f4523e5" (UID: "4e56b4af-b11d-4774-962c-3f7a9f4523e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:39:20.688633 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.688616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e56b4af-b11d-4774-962c-3f7a9f4523e5" (UID: "4e56b4af-b11d-4774-962c-3f7a9f4523e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:39:20.688702 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.688630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e56b4af-b11d-4774-962c-3f7a9f4523e5-kube-api-access-ggrvp" (OuterVolumeSpecName: "kube-api-access-ggrvp") pod "4e56b4af-b11d-4774-962c-3f7a9f4523e5" (UID: "4e56b4af-b11d-4774-962c-3f7a9f4523e5"). InnerVolumeSpecName "kube-api-access-ggrvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:39:20.787862 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.787842    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:20.787862 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.787862    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggrvp\" (UniqueName: \"kubernetes.io/projected/4e56b4af-b11d-4774-962c-3f7a9f4523e5-kube-api-access-ggrvp\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:20.787983 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.787871    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:20.787983 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.787880    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e56b4af-b11d-4774-962c-3f7a9f4523e5-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:20.961163 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.961134    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"]
Sep 15 21:39:20.965060 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:20.965036    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/auth-test-model-f36399-kserve-98856b67f-ththb"]
Sep 15 21:39:21.716293 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:21.716259    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k" podUID="ffa96195-e8d7-42a5-bac8-ba1502d71b49" containerName="main" containerID="cri-o://59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00" gracePeriod=45
Sep 15 21:39:21.972787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:21.972255    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:39:22.099301 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099273    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-model-cache\") pod \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") "
Sep 15 21:39:22.099528 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099394    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-home\") pod \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") "
Sep 15 21:39:22.099528 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099452    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cbqm\" (UniqueName: \"kubernetes.io/projected/ffa96195-e8d7-42a5-bac8-ba1502d71b49-kube-api-access-9cbqm\") pod \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") "
Sep 15 21:39:22.099528 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099515    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tmp-dir\") pod \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") "
Sep 15 21:39:22.099697 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-dshm\") pod \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") "
Sep 15 21:39:22.099697 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffa96195-e8d7-42a5-bac8-ba1502d71b49" (UID: "ffa96195-e8d7-42a5-bac8-ba1502d71b49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:39:22.099697 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099583    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tls-certs\") pod \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\" (UID: \"ffa96195-e8d7-42a5-bac8-ba1502d71b49\") "
Sep 15 21:39:22.099850 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-home" (OuterVolumeSpecName: "home") pod "ffa96195-e8d7-42a5-bac8-ba1502d71b49" (UID: "ffa96195-e8d7-42a5-bac8-ba1502d71b49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:39:22.099850 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099775    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffa96195-e8d7-42a5-bac8-ba1502d71b49" (UID: "ffa96195-e8d7-42a5-bac8-ba1502d71b49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:39:22.099951 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099901    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:22.099951 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099932    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:22.099951 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.099942    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:22.101703 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.101672    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-dshm" (OuterVolumeSpecName: "dshm") pod "ffa96195-e8d7-42a5-bac8-ba1502d71b49" (UID: "ffa96195-e8d7-42a5-bac8-ba1502d71b49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:39:22.101821 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.101707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa96195-e8d7-42a5-bac8-ba1502d71b49-kube-api-access-9cbqm" (OuterVolumeSpecName: "kube-api-access-9cbqm") pod "ffa96195-e8d7-42a5-bac8-ba1502d71b49" (UID: "ffa96195-e8d7-42a5-bac8-ba1502d71b49"). InnerVolumeSpecName "kube-api-access-9cbqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:39:22.101821 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.101731    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffa96195-e8d7-42a5-bac8-ba1502d71b49" (UID: "ffa96195-e8d7-42a5-bac8-ba1502d71b49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:39:22.201008 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.200965    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cbqm\" (UniqueName: \"kubernetes.io/projected/ffa96195-e8d7-42a5-bac8-ba1502d71b49-kube-api-access-9cbqm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:22.201008 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.201004    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa96195-e8d7-42a5-bac8-ba1502d71b49-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:22.201008 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.201014    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa96195-e8d7-42a5-bac8-ba1502d71b49-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:39:22.525920 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.525886    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e56b4af-b11d-4774-962c-3f7a9f4523e5" path="/var/lib/kubelet/pods/4e56b4af-b11d-4774-962c-3f7a9f4523e5/volumes"
Sep 15 21:39:22.651054 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.651025    2563 generic.go:356] "Generic (PLEG): container finished" podID="ffa96195-e8d7-42a5-bac8-ba1502d71b49" containerID="59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00" exitCode=0
Sep 15 21:39:22.651201 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.651070    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"
Sep 15 21:39:22.651201 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.651113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k" event={"ID":"ffa96195-e8d7-42a5-bac8-ba1502d71b49","Type":"ContainerDied","Data":"59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00"}
Sep 15 21:39:22.651201 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.651162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k" event={"ID":"ffa96195-e8d7-42a5-bac8-ba1502d71b49","Type":"ContainerDied","Data":"c0861df4fdcb35a8f508e40385503602781ed05c7260e66ce1e81202da29ec4b"}
Sep 15 21:39:22.651201 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.651184    2563 scope.go:117] "RemoveContainer" containerID="59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00"
Sep 15 21:39:22.660292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.660275    2563 scope.go:117] "RemoveContainer" containerID="59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00"
Sep 15 21:39:22.660567 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:39:22.660548    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00\": container with ID starting with 59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00 not found: ID does not exist" containerID="59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00"
Sep 15 21:39:22.660647 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.660578    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00"} err="failed to get container status \"59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00\": rpc error: code = NotFound desc = could not find container \"59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00\": container with ID starting with 59251fa4887d974cd2ef5b6fbc87ea51f0b6ce2ef3a0fa392a26826c52785e00 not found: ID does not exist"
Sep 15 21:39:22.667039 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.667016    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"]
Sep 15 21:39:22.671223 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:22.671203    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/auth-test-model-745dff-kserve-55666669ff-57f5k"]
Sep 15 21:39:24.526250 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:39:24.526217    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa96195-e8d7-42a5-bac8-ba1502d71b49" path="/var/lib/kubelet/pods/ffa96195-e8d7-42a5-bac8-ba1502d71b49/volumes"
Sep 15 21:40:10.518028 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.517999    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"]
Sep 15 21:40:10.518619 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518598    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2fef8e-d640-45b5-a100-a3d36cec5083" containerName="main"
Sep 15 21:40:10.518709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518621    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2fef8e-d640-45b5-a100-a3d36cec5083" containerName="main"
Sep 15 21:40:10.518709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518643    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af87cba1-663c-432f-adf7-bf25a39bf502" containerName="payload-pre-processing"
Sep 15 21:40:10.518709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518651    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af87cba1-663c-432f-adf7-bf25a39bf502" containerName="payload-pre-processing"
Sep 15 21:40:10.518709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518665    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" containerName="main"
Sep 15 21:40:10.518709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518673    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" containerName="main"
Sep 15 21:40:10.518709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518683    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e56b4af-b11d-4774-962c-3f7a9f4523e5" containerName="main"
Sep 15 21:40:10.518709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518691    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e56b4af-b11d-4774-962c-3f7a9f4523e5" containerName="main"
Sep 15 21:40:10.518709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518703    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa96195-e8d7-42a5-bac8-ba1502d71b49" containerName="main"
Sep 15 21:40:10.518709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518711    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa96195-e8d7-42a5-bac8-ba1502d71b49" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518721    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518728    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518738    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e073166-6091-4604-a4a6-df44b0a509e5" containerName="maas-api"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518746    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e073166-6091-4604-a4a6-df44b0a509e5" containerName="maas-api"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518768    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f170c4b-dff1-44fc-8081-4a7785446f59" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518775    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f170c4b-dff1-44fc-8081-4a7785446f59" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518786    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae15f1b9-db5b-4f74-b8ee-61cf58147429" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518794    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae15f1b9-db5b-4f74-b8ee-61cf58147429" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518807    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518816    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518824    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f87d01-75d9-4b23-82e5-e6750f6fefc9" containerName="maas-api"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518833    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f87d01-75d9-4b23-82e5-e6750f6fefc9" containerName="maas-api"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518935    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0c7e985-13f4-44f6-ada0-8aae1c5cedcb" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518945    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2fef8e-d640-45b5-a100-a3d36cec5083" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518955    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ff16e5-0dcb-4bd0-aa3d-0661c36746aa" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518964    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae15f1b9-db5b-4f74-b8ee-61cf58147429" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518974    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e56b4af-b11d-4774-962c-3f7a9f4523e5" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518983    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa96195-e8d7-42a5-bac8-ba1502d71b49" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.518991    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f87d01-75d9-4b23-82e5-e6750f6fefc9" containerName="maas-api"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.519001    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e32eff9-bb23-4daa-8cc3-f5e3ae09488d" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.519009    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f170c4b-dff1-44fc-8081-4a7785446f59" containerName="main"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.519019    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af87cba1-663c-432f-adf7-bf25a39bf502" containerName="payload-pre-processing"
Sep 15 21:40:10.519145 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.519028    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e073166-6091-4604-a4a6-df44b0a509e5" containerName="maas-api"
Sep 15 21:40:10.522800 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.522775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.525559 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.525530    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"default-dockercfg-clz6r\""
Sep 15 21:40:10.525773 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.525751    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"kube-root-ca.crt\""
Sep 15 21:40:10.525887 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.525794    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"sub-test-model-745dff-kserve-self-signed-certs\""
Sep 15 21:40:10.525962 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.525759    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"openshift-service-ca.crt\""
Sep 15 21:40:10.532230 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.531585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"]
Sep 15 21:40:10.591380 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.591325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cd012e-40c1-4773-b324-5c2cad2fe8af-tls-certs\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.591493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.591432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-model-cache\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.591493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.591473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-home\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.591571 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.591496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-tmp-dir\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.591571 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.591544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxpbg\" (UniqueName: \"kubernetes.io/projected/07cd012e-40c1-4773-b324-5c2cad2fe8af-kube-api-access-cxpbg\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.591636 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.591612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-dshm\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.692043 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.692017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-dshm\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.692179 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.692090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cd012e-40c1-4773-b324-5c2cad2fe8af-tls-certs\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.692179 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.692138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-model-cache\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.692301 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.692182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-home\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.692301 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.692212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-tmp-dir\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.692301 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.692274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxpbg\" (UniqueName: \"kubernetes.io/projected/07cd012e-40c1-4773-b324-5c2cad2fe8af-kube-api-access-cxpbg\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.692589 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.692563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-home\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.692707 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.692636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-model-cache\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.692707 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.692632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-tmp-dir\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.694680 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.694649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-dshm\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.694892 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.694867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cd012e-40c1-4773-b324-5c2cad2fe8af-tls-certs\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.700485 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.700461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxpbg\" (UniqueName: \"kubernetes.io/projected/07cd012e-40c1-4773-b324-5c2cad2fe8af-kube-api-access-cxpbg\") pod \"sub-test-model-745dff-kserve-66694db4cf-nnfqg\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") " pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:10.834819 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:10.834757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:11.107737 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:11.107706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"]
Sep 15 21:40:11.108519 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:40:11.108490    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07cd012e_40c1_4773_b324_5c2cad2fe8af.slice/crio-a1e79f0a741efe1367e7f2504657d8f8593e9c2a4c30dd732aa32fb92fc5c859 WatchSource:0}: Error finding container a1e79f0a741efe1367e7f2504657d8f8593e9c2a4c30dd732aa32fb92fc5c859: Status 404 returned error can't find the container with id a1e79f0a741efe1367e7f2504657d8f8593e9c2a4c30dd732aa32fb92fc5c859
Sep 15 21:40:11.927633 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:11.927603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg" event={"ID":"07cd012e-40c1-4773-b324-5c2cad2fe8af","Type":"ContainerStarted","Data":"fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c"}
Sep 15 21:40:11.927992 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:11.927641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg" event={"ID":"07cd012e-40c1-4773-b324-5c2cad2fe8af","Type":"ContainerStarted","Data":"a1e79f0a741efe1367e7f2504657d8f8593e9c2a4c30dd732aa32fb92fc5c859"}
Sep 15 21:40:11.945653 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:11.945600    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg" podStartSLOduration=1.9455850319999999 podStartE2EDuration="1.945585032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:11.94293619 +0000 UTC m=+1801.905993562" watchObservedRunningTime="2026-09-15 21:40:11.945585032 +0000 UTC m=+1801.908642412"
Sep 15 21:40:20.835770 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:20.835741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:20.836148 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:20.835781    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:20.847978 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:20.847954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:20.993783 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:20.993758    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:40:54.687003 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:40:54.686966    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"]
Sep 15 21:41:00.210139 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.210109    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"]
Sep 15 21:41:00.213709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.213665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.216453 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.216426    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"openshift-service-ca.crt\""
Sep 15 21:41:00.216453 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.216426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"rate-test-model-f36399-kserve-self-signed-certs\""
Sep 15 21:41:00.216453 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.216449    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"default-dockercfg-cmswh\""
Sep 15 21:41:00.216725 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.216500    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"kube-root-ca.crt\""
Sep 15 21:41:00.223676 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.223652    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"]
Sep 15 21:41:00.290406 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.290377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-dshm\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.290580 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.290417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-tmp-dir\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.290580 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.290443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4feb820-f343-49ca-80c6-59ca716fc033-tls-certs\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.290580 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.290476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-model-cache\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.290580 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.290499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkmjm\" (UniqueName: \"kubernetes.io/projected/d4feb820-f343-49ca-80c6-59ca716fc033-kube-api-access-bkmjm\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.290787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.290610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-home\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.392194 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.392168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-dshm\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.392327 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.392201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-tmp-dir\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.392327 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.392218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4feb820-f343-49ca-80c6-59ca716fc033-tls-certs\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.392327 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.392239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-model-cache\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.392492 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.392376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkmjm\" (UniqueName: \"kubernetes.io/projected/d4feb820-f343-49ca-80c6-59ca716fc033-kube-api-access-bkmjm\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.392492 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.392416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-home\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.392717 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.392698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-tmp-dir\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.392812 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.392750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-home\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.392812 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.392782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-model-cache\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.394516 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.394496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-dshm\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.394813 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.394794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4feb820-f343-49ca-80c6-59ca716fc033-tls-certs\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.399985 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.399957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkmjm\" (UniqueName: \"kubernetes.io/projected/d4feb820-f343-49ca-80c6-59ca716fc033-kube-api-access-bkmjm\") pod \"rate-test-model-f36399-kserve-8555c4d979-cjdph\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") " pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.524510 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.524442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:00.792963 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:00.792938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"]
Sep 15 21:41:00.794601 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:41:00.794571    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4feb820_f343_49ca_80c6_59ca716fc033.slice/crio-ec19346a817b23db18751cd3edeea62c2849d51ff4fef11a88d7f5a54d5d2df4 WatchSource:0}: Error finding container ec19346a817b23db18751cd3edeea62c2849d51ff4fef11a88d7f5a54d5d2df4: Status 404 returned error can't find the container with id ec19346a817b23db18751cd3edeea62c2849d51ff4fef11a88d7f5a54d5d2df4
Sep 15 21:41:01.221537 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:01.221504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph" event={"ID":"d4feb820-f343-49ca-80c6-59ca716fc033","Type":"ContainerStarted","Data":"c8cd05d2de37fc225180c891dcdc8c04b8d1352cf2a6ce5132f70bd4a83613e6"}
Sep 15 21:41:01.221870 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:01.221550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph" event={"ID":"d4feb820-f343-49ca-80c6-59ca716fc033","Type":"ContainerStarted","Data":"ec19346a817b23db18751cd3edeea62c2849d51ff4fef11a88d7f5a54d5d2df4"}
Sep 15 21:41:01.238623 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:01.238582    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph" podStartSLOduration=1.238569282 podStartE2EDuration="1.238569282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:01.235886886 +0000 UTC m=+1851.198944266" watchObservedRunningTime="2026-09-15 21:41:01.238569282 +0000 UTC m=+1851.201626662"
Sep 15 21:41:09.705684 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:09.705645    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg" podUID="07cd012e-40c1-4773-b324-5c2cad2fe8af" containerName="main" containerID="cri-o://fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c" gracePeriod=45
Sep 15 21:41:09.949233 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:09.949209    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:41:10.084396 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084298    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-tmp-dir\") pod \"07cd012e-40c1-4773-b324-5c2cad2fe8af\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") "
Sep 15 21:41:10.084396 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084334    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-model-cache\") pod \"07cd012e-40c1-4773-b324-5c2cad2fe8af\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") "
Sep 15 21:41:10.084621 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084405    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-dshm\") pod \"07cd012e-40c1-4773-b324-5c2cad2fe8af\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") "
Sep 15 21:41:10.084621 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084429    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxpbg\" (UniqueName: \"kubernetes.io/projected/07cd012e-40c1-4773-b324-5c2cad2fe8af-kube-api-access-cxpbg\") pod \"07cd012e-40c1-4773-b324-5c2cad2fe8af\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") "
Sep 15 21:41:10.084621 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cd012e-40c1-4773-b324-5c2cad2fe8af-tls-certs\") pod \"07cd012e-40c1-4773-b324-5c2cad2fe8af\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") "
Sep 15 21:41:10.084621 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084480    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-home\") pod \"07cd012e-40c1-4773-b324-5c2cad2fe8af\" (UID: \"07cd012e-40c1-4773-b324-5c2cad2fe8af\") "
Sep 15 21:41:10.084621 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084598    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07cd012e-40c1-4773-b324-5c2cad2fe8af" (UID: "07cd012e-40c1-4773-b324-5c2cad2fe8af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:10.084855 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084667    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-model-cache" (OuterVolumeSpecName: "model-cache") pod "07cd012e-40c1-4773-b324-5c2cad2fe8af" (UID: "07cd012e-40c1-4773-b324-5c2cad2fe8af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:10.084913 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084894    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:41:10.084955 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084920    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:41:10.084955 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.084897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-home" (OuterVolumeSpecName: "home") pod "07cd012e-40c1-4773-b324-5c2cad2fe8af" (UID: "07cd012e-40c1-4773-b324-5c2cad2fe8af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:10.086574 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.086551    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-dshm" (OuterVolumeSpecName: "dshm") pod "07cd012e-40c1-4773-b324-5c2cad2fe8af" (UID: "07cd012e-40c1-4773-b324-5c2cad2fe8af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:10.086574 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.086558    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07cd012e-40c1-4773-b324-5c2cad2fe8af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07cd012e-40c1-4773-b324-5c2cad2fe8af" (UID: "07cd012e-40c1-4773-b324-5c2cad2fe8af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:41:10.086679 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.086651    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07cd012e-40c1-4773-b324-5c2cad2fe8af-kube-api-access-cxpbg" (OuterVolumeSpecName: "kube-api-access-cxpbg") pod "07cd012e-40c1-4773-b324-5c2cad2fe8af" (UID: "07cd012e-40c1-4773-b324-5c2cad2fe8af"). InnerVolumeSpecName "kube-api-access-cxpbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:41:10.185504 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.185473    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:41:10.185504 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.185498    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxpbg\" (UniqueName: \"kubernetes.io/projected/07cd012e-40c1-4773-b324-5c2cad2fe8af-kube-api-access-cxpbg\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:41:10.185504 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.185508    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cd012e-40c1-4773-b324-5c2cad2fe8af-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:41:10.185727 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.185517    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cd012e-40c1-4773-b324-5c2cad2fe8af-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:41:10.274992 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.274961    2563 generic.go:356] "Generic (PLEG): container finished" podID="07cd012e-40c1-4773-b324-5c2cad2fe8af" containerID="fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c" exitCode=0
Sep 15 21:41:10.275142 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.275010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"
Sep 15 21:41:10.275142 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.275037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg" event={"ID":"07cd012e-40c1-4773-b324-5c2cad2fe8af","Type":"ContainerDied","Data":"fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c"}
Sep 15 21:41:10.275142 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.275087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg" event={"ID":"07cd012e-40c1-4773-b324-5c2cad2fe8af","Type":"ContainerDied","Data":"a1e79f0a741efe1367e7f2504657d8f8593e9c2a4c30dd732aa32fb92fc5c859"}
Sep 15 21:41:10.275142 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.275104    2563 scope.go:117] "RemoveContainer" containerID="fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c"
Sep 15 21:41:10.284805 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.284788    2563 scope.go:117] "RemoveContainer" containerID="fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c"
Sep 15 21:41:10.285057 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:41:10.285040    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c\": container with ID starting with fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c not found: ID does not exist" containerID="fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c"
Sep 15 21:41:10.285105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.285065    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c"} err="failed to get container status \"fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c\": rpc error: code = NotFound desc = could not find container \"fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c\": container with ID starting with fe64f46bd117facf9a9bc3a6e8146964818b52174a561e93469375741782c62c not found: ID does not exist"
Sep 15 21:41:10.297011 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.296989    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"]
Sep 15 21:41:10.301199 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.301179    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/sub-test-model-745dff-kserve-66694db4cf-nnfqg"]
Sep 15 21:41:10.527131 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.527101    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07cd012e-40c1-4773-b324-5c2cad2fe8af" path="/var/lib/kubelet/pods/07cd012e-40c1-4773-b324-5c2cad2fe8af/volumes"
Sep 15 21:41:10.527407 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.527393    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:10.527460 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.527425    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:10.536656 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:10.536634    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:41:11.291127 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:41:11.291044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:42:01.392767 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:01.392729    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"]
Sep 15 21:42:16.412221 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.412165    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph" podUID="d4feb820-f343-49ca-80c6-59ca716fc033" containerName="main" containerID="cri-o://c8cd05d2de37fc225180c891dcdc8c04b8d1352cf2a6ce5132f70bd4a83613e6" gracePeriod=45
Sep 15 21:42:16.660117 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.660086    2563 generic.go:356] "Generic (PLEG): container finished" podID="d4feb820-f343-49ca-80c6-59ca716fc033" containerID="c8cd05d2de37fc225180c891dcdc8c04b8d1352cf2a6ce5132f70bd4a83613e6" exitCode=0
Sep 15 21:42:16.660273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.660153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph" event={"ID":"d4feb820-f343-49ca-80c6-59ca716fc033","Type":"ContainerDied","Data":"c8cd05d2de37fc225180c891dcdc8c04b8d1352cf2a6ce5132f70bd4a83613e6"}
Sep 15 21:42:16.660273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.660191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph" event={"ID":"d4feb820-f343-49ca-80c6-59ca716fc033","Type":"ContainerDied","Data":"ec19346a817b23db18751cd3edeea62c2849d51ff4fef11a88d7f5a54d5d2df4"}
Sep 15 21:42:16.660273 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.660202    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec19346a817b23db18751cd3edeea62c2849d51ff4fef11a88d7f5a54d5d2df4"
Sep 15 21:42:16.661747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.661733    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:42:16.772140 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-home\") pod \"d4feb820-f343-49ca-80c6-59ca716fc033\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") "
Sep 15 21:42:16.772140 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772083    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-model-cache\") pod \"d4feb820-f343-49ca-80c6-59ca716fc033\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") "
Sep 15 21:42:16.772140 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-dshm\") pod \"d4feb820-f343-49ca-80c6-59ca716fc033\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") "
Sep 15 21:42:16.772493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772168    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4feb820-f343-49ca-80c6-59ca716fc033-tls-certs\") pod \"d4feb820-f343-49ca-80c6-59ca716fc033\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") "
Sep 15 21:42:16.772493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772239    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkmjm\" (UniqueName: \"kubernetes.io/projected/d4feb820-f343-49ca-80c6-59ca716fc033-kube-api-access-bkmjm\") pod \"d4feb820-f343-49ca-80c6-59ca716fc033\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") "
Sep 15 21:42:16.772493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-tmp-dir\") pod \"d4feb820-f343-49ca-80c6-59ca716fc033\" (UID: \"d4feb820-f343-49ca-80c6-59ca716fc033\") "
Sep 15 21:42:16.772493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4feb820-f343-49ca-80c6-59ca716fc033" (UID: "d4feb820-f343-49ca-80c6-59ca716fc033"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:16.772493 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-home" (OuterVolumeSpecName: "home") pod "d4feb820-f343-49ca-80c6-59ca716fc033" (UID: "d4feb820-f343-49ca-80c6-59ca716fc033"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:16.772744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4feb820-f343-49ca-80c6-59ca716fc033" (UID: "d4feb820-f343-49ca-80c6-59ca716fc033"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:16.772744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772700    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:42:16.772744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772714    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:42:16.772744 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.772744    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:42:16.774398 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.774369    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4feb820-f343-49ca-80c6-59ca716fc033-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4feb820-f343-49ca-80c6-59ca716fc033" (UID: "d4feb820-f343-49ca-80c6-59ca716fc033"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:42:16.774500 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.774411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4feb820-f343-49ca-80c6-59ca716fc033-kube-api-access-bkmjm" (OuterVolumeSpecName: "kube-api-access-bkmjm") pod "d4feb820-f343-49ca-80c6-59ca716fc033" (UID: "d4feb820-f343-49ca-80c6-59ca716fc033"). InnerVolumeSpecName "kube-api-access-bkmjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:42:16.774500 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.774437    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-dshm" (OuterVolumeSpecName: "dshm") pod "d4feb820-f343-49ca-80c6-59ca716fc033" (UID: "d4feb820-f343-49ca-80c6-59ca716fc033"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:16.874051 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.874011    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4feb820-f343-49ca-80c6-59ca716fc033-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:42:16.874051 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.874046    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkmjm\" (UniqueName: \"kubernetes.io/projected/d4feb820-f343-49ca-80c6-59ca716fc033-kube-api-access-bkmjm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:42:16.874051 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:16.874057    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4feb820-f343-49ca-80c6-59ca716fc033-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:42:17.667394 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:17.667362    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"
Sep 15 21:42:17.696230 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:17.696199    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"]
Sep 15 21:42:17.713139 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:17.713105    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/rate-test-model-f36399-kserve-8555c4d979-cjdph"]
Sep 15 21:42:18.525687 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:42:18.525651    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4feb820-f343-49ca-80c6-59ca716fc033" path="/var/lib/kubelet/pods/d4feb820-f343-49ca-80c6-59ca716fc033/volumes"
Sep 15 21:43:11.693016 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:11.692935    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-q7g9m_5aab6349-f631-40d4-8e1b-6ad8fbab3f7a/payload-processing/0.log"
Sep 15 21:43:15.511453 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.511420    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"]
Sep 15 21:43:15.511842 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.511828    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07cd012e-40c1-4773-b324-5c2cad2fe8af" containerName="main"
Sep 15 21:43:15.511902 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.511845    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07cd012e-40c1-4773-b324-5c2cad2fe8af" containerName="main"
Sep 15 21:43:15.511902 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.511854    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4feb820-f343-49ca-80c6-59ca716fc033" containerName="main"
Sep 15 21:43:15.511902 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.511860    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4feb820-f343-49ca-80c6-59ca716fc033" containerName="main"
Sep 15 21:43:15.512004 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.511921    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4feb820-f343-49ca-80c6-59ca716fc033" containerName="main"
Sep 15 21:43:15.512004 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.511932    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="07cd012e-40c1-4773-b324-5c2cad2fe8af" containerName="main"
Sep 15 21:43:15.514709 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.514694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.517590 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.517559    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6d9d39\"/\"kube-root-ca.crt\""
Sep 15 21:43:15.517733 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.517600    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6d9d39\"/\"openshift-service-ca.crt\""
Sep 15 21:43:15.517733 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.517661    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6d9d39\"/\"ipp-route-6d9d39-kserve-self-signed-certs\""
Sep 15 21:43:15.517733 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.517687    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6d9d39\"/\"default-dockercfg-qrk9f\""
Sep 15 21:43:15.522339 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.522183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"]
Sep 15 21:43:15.703782 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.703757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-dshm\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.703896 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.703789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-model-cache\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.703896 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.703806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tls-certs\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.703973 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.703897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bmxj\" (UniqueName: \"kubernetes.io/projected/31fe6f82-cebc-4780-8d26-837bfdc97bc2-kube-api-access-4bmxj\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.703973 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.703926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tmp-dir\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.703973 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.703961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-home\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.805286 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.805210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-dshm\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.805286 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.805256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-model-cache\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.805286 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.805271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tls-certs\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.805568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.805312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bmxj\" (UniqueName: \"kubernetes.io/projected/31fe6f82-cebc-4780-8d26-837bfdc97bc2-kube-api-access-4bmxj\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.805568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.805330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tmp-dir\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.805568 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.805382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-home\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.805734 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.805689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-model-cache\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.805782 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.805739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-home\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.805782 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.805761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tmp-dir\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.807416 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.807395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-dshm\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.807673 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.807657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tls-certs\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.812832 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.812810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bmxj\" (UniqueName: \"kubernetes.io/projected/31fe6f82-cebc-4780-8d26-837bfdc97bc2-kube-api-access-4bmxj\") pod \"ipp-route-6d9d39-kserve-6c978ff955-fttnk\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") " pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:15.824685 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:15.824664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:16.095206 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:16.095172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"]
Sep 15 21:43:16.095934 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:43:16.095906    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31fe6f82_cebc_4780_8d26_837bfdc97bc2.slice/crio-162bbab834904487679642b1135245963e0ffe200d0e2ad15dc19b3b7c088d08 WatchSource:0}: Error finding container 162bbab834904487679642b1135245963e0ffe200d0e2ad15dc19b3b7c088d08: Status 404 returned error can't find the container with id 162bbab834904487679642b1135245963e0ffe200d0e2ad15dc19b3b7c088d08
Sep 15 21:43:17.002293 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:17.002255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk" event={"ID":"31fe6f82-cebc-4780-8d26-837bfdc97bc2","Type":"ContainerStarted","Data":"e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9"}
Sep 15 21:43:17.002819 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:17.002301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk" event={"ID":"31fe6f82-cebc-4780-8d26-837bfdc97bc2","Type":"ContainerStarted","Data":"162bbab834904487679642b1135245963e0ffe200d0e2ad15dc19b3b7c088d08"}
Sep 15 21:43:17.023166 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:17.023119    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk" podStartSLOduration=2.0231052370000002 podStartE2EDuration="2.023105237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:17.020948387 +0000 UTC m=+1986.984005768" watchObservedRunningTime="2026-09-15 21:43:17.023105237 +0000 UTC m=+1986.986162617"
Sep 15 21:43:25.825663 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:25.825625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:25.826043 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:25.825681    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:25.837856 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:25.837832    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:26.066663 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:26.066638    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:43:37.524779 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:37.524745    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-q7g9m_5aab6349-f631-40d4-8e1b-6ad8fbab3f7a/payload-processing/0.log"
Sep 15 21:43:44.039172 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.039140    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"]
Sep 15 21:43:44.042981 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.042961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.045131 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.045113    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-a36624-serving-cert\""
Sep 15 21:43:44.054339 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.054317    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"]
Sep 15 21:43:44.166397 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.166337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.166397 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.166395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.166609 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.166419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqpv\" (UniqueName: \"kubernetes.io/projected/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-kube-api-access-wxqpv\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.166609 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.166485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.267153 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.267117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.267374 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.267200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.267374 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.267221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.267374 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.267236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqpv\" (UniqueName: \"kubernetes.io/projected/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-kube-api-access-wxqpv\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.267948 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.267924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.269579 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.269561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.269742 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.269722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.275929 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.275904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqpv\" (UniqueName: \"kubernetes.io/projected/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-kube-api-access-wxqpv\") pod \"maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.355658 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.355579    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:44.628968 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.628941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"]
Sep 15 21:43:44.630557 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:43:44.630526    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e15d8a_9dc0_4a12_ae76_f868df11e9cc.slice/crio-e850194318ace6a101977a787f59f29e9b1abc36b4cb152b005f01c5c0d4f8a3 WatchSource:0}: Error finding container e850194318ace6a101977a787f59f29e9b1abc36b4cb152b005f01c5c0d4f8a3: Status 404 returned error can't find the container with id e850194318ace6a101977a787f59f29e9b1abc36b4cb152b005f01c5c0d4f8a3
Sep 15 21:43:44.631875 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:44.631858    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:43:45.168701 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:45.168678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt" event={"ID":"68e15d8a-9dc0-4a12-ae76-f868df11e9cc","Type":"ContainerStarted","Data":"3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c"}
Sep 15 21:43:45.169006 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:45.168728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt" event={"ID":"68e15d8a-9dc0-4a12-ae76-f868df11e9cc","Type":"ContainerStarted","Data":"e850194318ace6a101977a787f59f29e9b1abc36b4cb152b005f01c5c0d4f8a3"}
Sep 15 21:43:45.169006 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:45.168756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:43:45.187848 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:45.187798    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt" podStartSLOduration=0.714390101 podStartE2EDuration="1.187780204s" totalImagesPullingTime="473.390103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:45.185579416 +0000 UTC m=+2015.148636797" watchObservedRunningTime="2026-09-15 21:43:45.187780204 +0000 UTC m=+2015.150837585"
Sep 15 21:43:51.180160 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:43:51.180132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:44:22.618877 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.618828    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"]
Sep 15 21:44:22.619484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.619086    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt" podUID="68e15d8a-9dc0-4a12-ae76-f868df11e9cc" containerName="maas-api" containerID="cri-o://3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c" gracePeriod=30
Sep 15 21:44:22.867295 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.867270    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:44:22.928737 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.928703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-openshift-service-ca\") pod \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") "
Sep 15 21:44:22.928922 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.928751    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxqpv\" (UniqueName: \"kubernetes.io/projected/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-kube-api-access-wxqpv\") pod \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") "
Sep 15 21:44:22.928922 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.928784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-metrics-certs\") pod \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") "
Sep 15 21:44:22.928922 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.928850    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-maas-api-tls\") pod \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\" (UID: \"68e15d8a-9dc0-4a12-ae76-f868df11e9cc\") "
Sep 15 21:44:22.929169 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.929146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68e15d8a-9dc0-4a12-ae76-f868df11e9cc" (UID: "68e15d8a-9dc0-4a12-ae76-f868df11e9cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:22.929260 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.929243    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-openshift-service-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:44:22.930974 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.930945    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68e15d8a-9dc0-4a12-ae76-f868df11e9cc" (UID: "68e15d8a-9dc0-4a12-ae76-f868df11e9cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:44:22.931101 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.930966    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68e15d8a-9dc0-4a12-ae76-f868df11e9cc" (UID: "68e15d8a-9dc0-4a12-ae76-f868df11e9cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:44:22.931101 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:22.931072    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-kube-api-access-wxqpv" (OuterVolumeSpecName: "kube-api-access-wxqpv") pod "68e15d8a-9dc0-4a12-ae76-f868df11e9cc" (UID: "68e15d8a-9dc0-4a12-ae76-f868df11e9cc"). InnerVolumeSpecName "kube-api-access-wxqpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:23.030447 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.030413    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxqpv\" (UniqueName: \"kubernetes.io/projected/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-kube-api-access-wxqpv\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:44:23.030447 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.030442    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-metrics-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:44:23.030447 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.030453    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e15d8a-9dc0-4a12-ae76-f868df11e9cc-maas-api-tls\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:44:23.390186 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.390150    2563 generic.go:356] "Generic (PLEG): container finished" podID="68e15d8a-9dc0-4a12-ae76-f868df11e9cc" containerID="3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c" exitCode=0
Sep 15 21:44:23.390484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.390190    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"
Sep 15 21:44:23.390484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.390230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt" event={"ID":"68e15d8a-9dc0-4a12-ae76-f868df11e9cc","Type":"ContainerDied","Data":"3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c"}
Sep 15 21:44:23.390484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.390291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt" event={"ID":"68e15d8a-9dc0-4a12-ae76-f868df11e9cc","Type":"ContainerDied","Data":"e850194318ace6a101977a787f59f29e9b1abc36b4cb152b005f01c5c0d4f8a3"}
Sep 15 21:44:23.390484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.390322    2563 scope.go:117] "RemoveContainer" containerID="3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c"
Sep 15 21:44:23.400684 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.400669    2563 scope.go:117] "RemoveContainer" containerID="3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c"
Sep 15 21:44:23.400961 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:44:23.400943    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c\": container with ID starting with 3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c not found: ID does not exist" containerID="3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c"
Sep 15 21:44:23.401010 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.400973    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c"} err="failed to get container status \"3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c\": rpc error: code = NotFound desc = could not find container \"3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c\": container with ID starting with 3551885839d31460b7cc7078f1d9a9eb25ac1f2a63bd22e94c9a6670e5fe8d7c not found: ID does not exist"
Sep 15 21:44:23.415533 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.415506    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"]
Sep 15 21:44:23.422378 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:23.422354    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a36624-759bdc5d54-c7lnt"]
Sep 15 21:44:24.527933 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:24.527890    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68e15d8a-9dc0-4a12-ae76-f868df11e9cc" path="/var/lib/kubelet/pods/68e15d8a-9dc0-4a12-ae76-f868df11e9cc/volumes"
Sep 15 21:44:54.334540 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:44:54.334505    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"]
Sep 15 21:45:00.142688 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:00.142655    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4"]
Sep 15 21:45:00.143100 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:00.143087    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e15d8a-9dc0-4a12-ae76-f868df11e9cc" containerName="maas-api"
Sep 15 21:45:00.143100 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:00.143101    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e15d8a-9dc0-4a12-ae76-f868df11e9cc" containerName="maas-api"
Sep 15 21:45:00.143178 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:00.143170    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="68e15d8a-9dc0-4a12-ae76-f868df11e9cc" containerName="maas-api"
Sep 15 21:45:00.146468 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:00.146444    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4"
Sep 15 21:45:00.148621 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:00.148601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:45:00.153200 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:00.153179    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4"]
Sep 15 21:45:00.158695 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:00.158676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4"
Sep 15 21:45:00.427025 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:00.426995    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4"]
Sep 15 21:45:00.428476 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:45:00.428453    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f47b18a_7224_44dc_bcc5_ec0269498fcf.slice/crio-0b10ce47f3b285ce699b942712141827c4b0790b45b82a39c632422e95319295 WatchSource:0}: Error finding container 0b10ce47f3b285ce699b942712141827c4b0790b45b82a39c632422e95319295: Status 404 returned error can't find the container with id 0b10ce47f3b285ce699b942712141827c4b0790b45b82a39c632422e95319295
Sep 15 21:45:00.612533 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:00.612499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4" event={"ID":"3f47b18a-7224-44dc-bcc5-ec0269498fcf","Type":"ContainerStarted","Data":"0b10ce47f3b285ce699b942712141827c4b0790b45b82a39c632422e95319295"}
Sep 15 21:45:01.619387 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:01.619358    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f47b18a-7224-44dc-bcc5-ec0269498fcf" containerID="3ecfecc46f2e67a04a40619573860d814cd738741f07f4b9b2d9ab7a96932922" exitCode=0
Sep 15 21:45:01.619804 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:01.619435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4" event={"ID":"3f47b18a-7224-44dc-bcc5-ec0269498fcf","Type":"ContainerDied","Data":"3ecfecc46f2e67a04a40619573860d814cd738741f07f4b9b2d9ab7a96932922"}
Sep 15 21:45:02.758701 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:02.758674    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4"
Sep 15 21:45:03.632939 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:03.632907    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4"
Sep 15 21:45:03.632939 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:03.632923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4" event={"ID":"3f47b18a-7224-44dc-bcc5-ec0269498fcf","Type":"ContainerDied","Data":"0b10ce47f3b285ce699b942712141827c4b0790b45b82a39c632422e95319295"}
Sep 15 21:45:03.633156 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:03.632964    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b10ce47f3b285ce699b942712141827c4b0790b45b82a39c632422e95319295"
Sep 15 21:45:09.353798 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.353761    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk" podUID="31fe6f82-cebc-4780-8d26-837bfdc97bc2" containerName="main" containerID="cri-o://e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9" gracePeriod=45
Sep 15 21:45:09.589534 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.589512    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:45:09.667738 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.667711    2563 generic.go:356] "Generic (PLEG): container finished" podID="31fe6f82-cebc-4780-8d26-837bfdc97bc2" containerID="e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9" exitCode=0
Sep 15 21:45:09.667913 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.667754    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"
Sep 15 21:45:09.667913 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.667785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk" event={"ID":"31fe6f82-cebc-4780-8d26-837bfdc97bc2","Type":"ContainerDied","Data":"e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9"}
Sep 15 21:45:09.667913 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.667827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk" event={"ID":"31fe6f82-cebc-4780-8d26-837bfdc97bc2","Type":"ContainerDied","Data":"162bbab834904487679642b1135245963e0ffe200d0e2ad15dc19b3b7c088d08"}
Sep 15 21:45:09.667913 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.667844    2563 scope.go:117] "RemoveContainer" containerID="e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9"
Sep 15 21:45:09.676435 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.676419    2563 scope.go:117] "RemoveContainer" containerID="e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9"
Sep 15 21:45:09.676689 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:45:09.676671    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9\": container with ID starting with e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9 not found: ID does not exist" containerID="e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9"
Sep 15 21:45:09.676740 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.676697    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9"} err="failed to get container status \"e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9\": rpc error: code = NotFound desc = could not find container \"e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9\": container with ID starting with e9c0211c1aa695228eb37f2caae37aa143dc622d2afb70a5ceeb1a99001877c9 not found: ID does not exist"
Sep 15 21:45:09.764892 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.764869    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tls-certs\") pod \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") "
Sep 15 21:45:09.765017 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.764933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-dshm\") pod \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") "
Sep 15 21:45:09.765017 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.764947    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tmp-dir\") pod \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") "
Sep 15 21:45:09.765017 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.764967    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-model-cache\") pod \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") "
Sep 15 21:45:09.765017 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.764994    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bmxj\" (UniqueName: \"kubernetes.io/projected/31fe6f82-cebc-4780-8d26-837bfdc97bc2-kube-api-access-4bmxj\") pod \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") "
Sep 15 21:45:09.765231 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.765046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-home\") pod \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\" (UID: \"31fe6f82-cebc-4780-8d26-837bfdc97bc2\") "
Sep 15 21:45:09.765286 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.765231    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31fe6f82-cebc-4780-8d26-837bfdc97bc2" (UID: "31fe6f82-cebc-4780-8d26-837bfdc97bc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:09.765286 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.765246    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "31fe6f82-cebc-4780-8d26-837bfdc97bc2" (UID: "31fe6f82-cebc-4780-8d26-837bfdc97bc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:09.765446 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.765423    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-home" (OuterVolumeSpecName: "home") pod "31fe6f82-cebc-4780-8d26-837bfdc97bc2" (UID: "31fe6f82-cebc-4780-8d26-837bfdc97bc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:09.765576 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.765553    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:45:09.765576 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.765576    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:45:09.767094 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.767073    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-dshm" (OuterVolumeSpecName: "dshm") pod "31fe6f82-cebc-4780-8d26-837bfdc97bc2" (UID: "31fe6f82-cebc-4780-8d26-837bfdc97bc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:09.767176 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.767084    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fe6f82-cebc-4780-8d26-837bfdc97bc2-kube-api-access-4bmxj" (OuterVolumeSpecName: "kube-api-access-4bmxj") pod "31fe6f82-cebc-4780-8d26-837bfdc97bc2" (UID: "31fe6f82-cebc-4780-8d26-837bfdc97bc2"). InnerVolumeSpecName "kube-api-access-4bmxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:09.767176 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.767078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31fe6f82-cebc-4780-8d26-837bfdc97bc2" (UID: "31fe6f82-cebc-4780-8d26-837bfdc97bc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:45:09.867021 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.866994    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:45:09.867021 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.867016    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31fe6f82-cebc-4780-8d26-837bfdc97bc2-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:45:09.867021 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.867024    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31fe6f82-cebc-4780-8d26-837bfdc97bc2-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:45:09.867222 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.867032    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bmxj\" (UniqueName: \"kubernetes.io/projected/31fe6f82-cebc-4780-8d26-837bfdc97bc2-kube-api-access-4bmxj\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:45:09.996525 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.996495    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"]
Sep 15 21:45:09.998879 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:09.998851    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6d9d39/ipp-route-6d9d39-kserve-6c978ff955-fttnk"]
Sep 15 21:45:10.526141 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:45:10.526108    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31fe6f82-cebc-4780-8d26-837bfdc97bc2" path="/var/lib/kubelet/pods/31fe6f82-cebc-4780-8d26-837bfdc97bc2/volumes"
Sep 15 21:46:06.336168 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.336078    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"]
Sep 15 21:46:06.341249 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.336377    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k" podUID="38926e29-d812-4502-89ba-3a7a01b6f54f" containerName="maas-api" containerID="cri-o://1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b" gracePeriod=30
Sep 15 21:46:06.364980 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.364954    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4"]
Sep 15 21:46:06.368679 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.368655    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f36399-29825145-jkfm4"]
Sep 15 21:46:06.527476 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.527450    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f47b18a-7224-44dc-bcc5-ec0269498fcf" path="/var/lib/kubelet/pods/3f47b18a-7224-44dc-bcc5-ec0269498fcf/volumes"
Sep 15 21:46:06.585484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.585464    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:46:06.747547 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.747517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkpbq\" (UniqueName: \"kubernetes.io/projected/38926e29-d812-4502-89ba-3a7a01b6f54f-kube-api-access-xkpbq\") pod \"38926e29-d812-4502-89ba-3a7a01b6f54f\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") "
Sep 15 21:46:06.747691 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.747571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-maas-api-tls\") pod \"38926e29-d812-4502-89ba-3a7a01b6f54f\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") "
Sep 15 21:46:06.747691 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.747599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38926e29-d812-4502-89ba-3a7a01b6f54f-openshift-service-ca\") pod \"38926e29-d812-4502-89ba-3a7a01b6f54f\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") "
Sep 15 21:46:06.747691 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.747614    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-metrics-certs\") pod \"38926e29-d812-4502-89ba-3a7a01b6f54f\" (UID: \"38926e29-d812-4502-89ba-3a7a01b6f54f\") "
Sep 15 21:46:06.747980 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.747953    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38926e29-d812-4502-89ba-3a7a01b6f54f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38926e29-d812-4502-89ba-3a7a01b6f54f" (UID: "38926e29-d812-4502-89ba-3a7a01b6f54f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:46:06.749874 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.749845    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38926e29-d812-4502-89ba-3a7a01b6f54f" (UID: "38926e29-d812-4502-89ba-3a7a01b6f54f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:46:06.749874 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.749863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "38926e29-d812-4502-89ba-3a7a01b6f54f" (UID: "38926e29-d812-4502-89ba-3a7a01b6f54f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:46:06.750027 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.749900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38926e29-d812-4502-89ba-3a7a01b6f54f-kube-api-access-xkpbq" (OuterVolumeSpecName: "kube-api-access-xkpbq") pod "38926e29-d812-4502-89ba-3a7a01b6f54f" (UID: "38926e29-d812-4502-89ba-3a7a01b6f54f"). InnerVolumeSpecName "kube-api-access-xkpbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:06.848292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.848263    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkpbq\" (UniqueName: \"kubernetes.io/projected/38926e29-d812-4502-89ba-3a7a01b6f54f-kube-api-access-xkpbq\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:46:06.848292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.848287    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-maas-api-tls\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:46:06.848292 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.848296    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38926e29-d812-4502-89ba-3a7a01b6f54f-openshift-service-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:46:06.848523 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.848305    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38926e29-d812-4502-89ba-3a7a01b6f54f-metrics-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:46:06.986204 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.986170    2563 generic.go:356] "Generic (PLEG): container finished" podID="38926e29-d812-4502-89ba-3a7a01b6f54f" containerID="1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b" exitCode=0
Sep 15 21:46:06.986373 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.986216    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"
Sep 15 21:46:06.986373 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.986251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k" event={"ID":"38926e29-d812-4502-89ba-3a7a01b6f54f","Type":"ContainerDied","Data":"1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b"}
Sep 15 21:46:06.986373 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.986305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k" event={"ID":"38926e29-d812-4502-89ba-3a7a01b6f54f","Type":"ContainerDied","Data":"1387fb12d3800196e4358b2884da4c0cd8a352fa1ca235921574da6b872ac4ff"}
Sep 15 21:46:06.986373 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.986327    2563 scope.go:117] "RemoveContainer" containerID="1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b"
Sep 15 21:46:06.996944 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.996926    2563 scope.go:117] "RemoveContainer" containerID="1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b"
Sep 15 21:46:06.997216 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:46:06.997199    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b\": container with ID starting with 1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b not found: ID does not exist" containerID="1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b"
Sep 15 21:46:06.997277 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:06.997229    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b"} err="failed to get container status \"1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b\": rpc error: code = NotFound desc = could not find container \"1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b\": container with ID starting with 1ce39cbbf2ae1c006b3b8462c2174ba8ddaac8a0b09e40a16a47b1ee3bf3079b not found: ID does not exist"
Sep 15 21:46:07.008376 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:07.008311    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"]
Sep 15 21:46:07.012028 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:07.012007    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f36399-6955c6cb48-zbx4k"]
Sep 15 21:46:08.526090 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:08.526059    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38926e29-d812-4502-89ba-3a7a01b6f54f" path="/var/lib/kubelet/pods/38926e29-d812-4502-89ba-3a7a01b6f54f/volumes"
Sep 15 21:46:33.647368 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:33.647301    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-8jc8g"]
Sep 15 21:46:33.647903 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:33.647557    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g" podUID="5c53dbcc-7d70-413b-ac17-797abca1f8df" containerName="manager" containerID="cri-o://2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21" gracePeriod=30
Sep 15 21:46:33.889447 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:33.889418    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:46:33.962396 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:33.962304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-metrics-certs\") pod \"5c53dbcc-7d70-413b-ac17-797abca1f8df\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") "
Sep 15 21:46:33.962396 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:33.962365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-webhook-cert\") pod \"5c53dbcc-7d70-413b-ac17-797abca1f8df\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") "
Sep 15 21:46:33.962566 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:33.962411    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq5l4\" (UniqueName: \"kubernetes.io/projected/5c53dbcc-7d70-413b-ac17-797abca1f8df-kube-api-access-tq5l4\") pod \"5c53dbcc-7d70-413b-ac17-797abca1f8df\" (UID: \"5c53dbcc-7d70-413b-ac17-797abca1f8df\") "
Sep 15 21:46:33.964318 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:33.964287    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5c53dbcc-7d70-413b-ac17-797abca1f8df" (UID: "5c53dbcc-7d70-413b-ac17-797abca1f8df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:46:33.964459 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:33.964367    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5c53dbcc-7d70-413b-ac17-797abca1f8df" (UID: "5c53dbcc-7d70-413b-ac17-797abca1f8df"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:46:33.964459 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:33.964372    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c53dbcc-7d70-413b-ac17-797abca1f8df-kube-api-access-tq5l4" (OuterVolumeSpecName: "kube-api-access-tq5l4") pod "5c53dbcc-7d70-413b-ac17-797abca1f8df" (UID: "5c53dbcc-7d70-413b-ac17-797abca1f8df"). InnerVolumeSpecName "kube-api-access-tq5l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:34.063085 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.063062    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-metrics-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:46:34.063085 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.063083    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c53dbcc-7d70-413b-ac17-797abca1f8df-webhook-cert\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:46:34.063218 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.063093    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq5l4\" (UniqueName: \"kubernetes.io/projected/5c53dbcc-7d70-413b-ac17-797abca1f8df-kube-api-access-tq5l4\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:46:34.151730 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.151702    2563 generic.go:356] "Generic (PLEG): container finished" podID="5c53dbcc-7d70-413b-ac17-797abca1f8df" containerID="2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21" exitCode=0
Sep 15 21:46:34.151833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.151754    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g"
Sep 15 21:46:34.151833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.151781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g" event={"ID":"5c53dbcc-7d70-413b-ac17-797abca1f8df","Type":"ContainerDied","Data":"2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21"}
Sep 15 21:46:34.151833 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.151830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f4d48d8b-8jc8g" event={"ID":"5c53dbcc-7d70-413b-ac17-797abca1f8df","Type":"ContainerDied","Data":"20133920e0be4bbde574ec3a78038f46f913ef565a205fb5fe285ddbbed8181d"}
Sep 15 21:46:34.151948 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.151852    2563 scope.go:117] "RemoveContainer" containerID="2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21"
Sep 15 21:46:34.161995 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.161978    2563 scope.go:117] "RemoveContainer" containerID="2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21"
Sep 15 21:46:34.162247 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:46:34.162230    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21\": container with ID starting with 2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21 not found: ID does not exist" containerID="2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21"
Sep 15 21:46:34.162296 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.162256    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21"} err="failed to get container status \"2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21\": rpc error: code = NotFound desc = could not find container \"2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21\": container with ID starting with 2e505fb58786d9c730addd1e9e14dad2398439b5919cdbd32160e1241d9c9e21 not found: ID does not exist"
Sep 15 21:46:34.173544 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.173524    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-8jc8g"]
Sep 15 21:46:34.177458 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.177440    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-8jc8g"]
Sep 15 21:46:34.525613 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:46:34.525589    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c53dbcc-7d70-413b-ac17-797abca1f8df" path="/var/lib/kubelet/pods/5c53dbcc-7d70-413b-ac17-797abca1f8df/volumes"
Sep 15 21:47:10.939420 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:47:10.939389    2563 scope.go:117] "RemoveContainer" containerID="c8cd05d2de37fc225180c891dcdc8c04b8d1352cf2a6ce5132f70bd4a83613e6"
Sep 15 21:51:07.399617 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.399583    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"]
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400001    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38926e29-d812-4502-89ba-3a7a01b6f54f" containerName="maas-api"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400013    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38926e29-d812-4502-89ba-3a7a01b6f54f" containerName="maas-api"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400028    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f47b18a-7224-44dc-bcc5-ec0269498fcf" containerName="cleanup"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400034    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f47b18a-7224-44dc-bcc5-ec0269498fcf" containerName="cleanup"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400048    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31fe6f82-cebc-4780-8d26-837bfdc97bc2" containerName="main"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400055    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31fe6f82-cebc-4780-8d26-837bfdc97bc2" containerName="main"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400070    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c53dbcc-7d70-413b-ac17-797abca1f8df" containerName="manager"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400076    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c53dbcc-7d70-413b-ac17-797abca1f8df" containerName="manager"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400137    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f47b18a-7224-44dc-bcc5-ec0269498fcf" containerName="cleanup"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400146    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c53dbcc-7d70-413b-ac17-797abca1f8df" containerName="manager"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400153    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="31fe6f82-cebc-4780-8d26-837bfdc97bc2" containerName="main"
Sep 15 21:51:07.402105 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.400159    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="38926e29-d812-4502-89ba-3a7a01b6f54f" containerName="maas-api"
Sep 15 21:51:07.403196 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.403177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.405574 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.405557    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f74dbafa-kserve-self-signed-certs\""
Sep 15 21:51:07.413264 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.413234    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"]
Sep 15 21:51:07.530911 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.530883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-model-cache\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.531043 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.530923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgg8t\" (UniqueName: \"kubernetes.io/projected/991244ae-a1ce-4c63-a5cc-efef8d29c806-kube-api-access-hgg8t\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.531043 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.530952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-dshm\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.531043 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.531023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-home\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.531187 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.531058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991244ae-a1ce-4c63-a5cc-efef8d29c806-tls-certs\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.531187 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.531081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-tmp-dir\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.632165 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.632133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-model-cache\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.632304 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.632205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgg8t\" (UniqueName: \"kubernetes.io/projected/991244ae-a1ce-4c63-a5cc-efef8d29c806-kube-api-access-hgg8t\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.632437 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.632412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-dshm\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.632536 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.632517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-home\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.632610 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.632593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-model-cache\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.632787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.632766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-home\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.632787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.632786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991244ae-a1ce-4c63-a5cc-efef8d29c806-tls-certs\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.632920 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.632843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-tmp-dir\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.633133 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.633112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-tmp-dir\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.634585 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.634565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-dshm\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.635005 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.634988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991244ae-a1ce-4c63-a5cc-efef8d29c806-tls-certs\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.639700 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.639672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgg8t\" (UniqueName: \"kubernetes.io/projected/991244ae-a1ce-4c63-a5cc-efef8d29c806-kube-api-access-hgg8t\") pod \"e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") " pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.712586 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.712524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:07.981529 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:07.981509    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"]
Sep 15 21:51:07.983386 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:51:07.983331    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod991244ae_a1ce_4c63_a5cc_efef8d29c806.slice/crio-d45c88a0f5dc8b98b32ce8d54e55b80c5e1b30e219856e6794ef0b2db6051e00 WatchSource:0}: Error finding container d45c88a0f5dc8b98b32ce8d54e55b80c5e1b30e219856e6794ef0b2db6051e00: Status 404 returned error can't find the container with id d45c88a0f5dc8b98b32ce8d54e55b80c5e1b30e219856e6794ef0b2db6051e00
Sep 15 21:51:08.676966 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:08.676929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb" event={"ID":"991244ae-a1ce-4c63-a5cc-efef8d29c806","Type":"ContainerStarted","Data":"241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81"}
Sep 15 21:51:08.676966 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:08.676971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb" event={"ID":"991244ae-a1ce-4c63-a5cc-efef8d29c806","Type":"ContainerStarted","Data":"d45c88a0f5dc8b98b32ce8d54e55b80c5e1b30e219856e6794ef0b2db6051e00"}
Sep 15 21:51:08.693143 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:08.693090    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb" podStartSLOduration=1.693069696 podStartE2EDuration="1.693069696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:51:08.691254931 +0000 UTC m=+2458.654312314" watchObservedRunningTime="2026-09-15 21:51:08.693069696 +0000 UTC m=+2458.656127077"
Sep 15 21:51:10.991821 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:10.991779    2563 scope.go:117] "RemoveContainer" containerID="3ecfecc46f2e67a04a40619573860d814cd738741f07f4b9b2d9ab7a96932922"
Sep 15 21:51:17.713291 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:17.713258    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:17.713694 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:17.713312    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:17.725918 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:17.725895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:17.738678 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:17.738657    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:51:50.387297 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:51:50.387266    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"]
Sep 15 21:52:05.406535 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.406446    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb" podUID="991244ae-a1ce-4c63-a5cc-efef8d29c806" containerName="main" containerID="cri-o://241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81" gracePeriod=45
Sep 15 21:52:05.651812 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.651787    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:52:05.800908 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.800851    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991244ae-a1ce-4c63-a5cc-efef8d29c806-tls-certs\") pod \"991244ae-a1ce-4c63-a5cc-efef8d29c806\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") "
Sep 15 21:52:05.800908 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.800881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgg8t\" (UniqueName: \"kubernetes.io/projected/991244ae-a1ce-4c63-a5cc-efef8d29c806-kube-api-access-hgg8t\") pod \"991244ae-a1ce-4c63-a5cc-efef8d29c806\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") "
Sep 15 21:52:05.801100 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.800914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-home\") pod \"991244ae-a1ce-4c63-a5cc-efef8d29c806\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") "
Sep 15 21:52:05.801100 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.800938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-tmp-dir\") pod \"991244ae-a1ce-4c63-a5cc-efef8d29c806\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") "
Sep 15 21:52:05.801100 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.800996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-dshm\") pod \"991244ae-a1ce-4c63-a5cc-efef8d29c806\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") "
Sep 15 21:52:05.801100 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.801075    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-model-cache\") pod \"991244ae-a1ce-4c63-a5cc-efef8d29c806\" (UID: \"991244ae-a1ce-4c63-a5cc-efef8d29c806\") "
Sep 15 21:52:05.801307 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.801161    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-home" (OuterVolumeSpecName: "home") pod "991244ae-a1ce-4c63-a5cc-efef8d29c806" (UID: "991244ae-a1ce-4c63-a5cc-efef8d29c806"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:52:05.801307 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.801236    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "991244ae-a1ce-4c63-a5cc-efef8d29c806" (UID: "991244ae-a1ce-4c63-a5cc-efef8d29c806"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:52:05.801435 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.801331    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-model-cache" (OuterVolumeSpecName: "model-cache") pod "991244ae-a1ce-4c63-a5cc-efef8d29c806" (UID: "991244ae-a1ce-4c63-a5cc-efef8d29c806"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:52:05.801484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.801450    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-model-cache\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:52:05.801484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.801466    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-home\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:52:05.801484 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.801478    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-tmp-dir\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:52:05.803003 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.802977    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991244ae-a1ce-4c63-a5cc-efef8d29c806-kube-api-access-hgg8t" (OuterVolumeSpecName: "kube-api-access-hgg8t") pod "991244ae-a1ce-4c63-a5cc-efef8d29c806" (UID: "991244ae-a1ce-4c63-a5cc-efef8d29c806"). InnerVolumeSpecName "kube-api-access-hgg8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:52:05.803396 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.803367    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991244ae-a1ce-4c63-a5cc-efef8d29c806-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "991244ae-a1ce-4c63-a5cc-efef8d29c806" (UID: "991244ae-a1ce-4c63-a5cc-efef8d29c806"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:52:05.803396 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.803372    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-dshm" (OuterVolumeSpecName: "dshm") pod "991244ae-a1ce-4c63-a5cc-efef8d29c806" (UID: "991244ae-a1ce-4c63-a5cc-efef8d29c806"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:52:05.902362 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.902327    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991244ae-a1ce-4c63-a5cc-efef8d29c806-dshm\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:52:05.902362 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.902358    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991244ae-a1ce-4c63-a5cc-efef8d29c806-tls-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:52:05.902481 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:05.902367    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgg8t\" (UniqueName: \"kubernetes.io/projected/991244ae-a1ce-4c63-a5cc-efef8d29c806-kube-api-access-hgg8t\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:52:06.004024 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:06.004001    2563 generic.go:356] "Generic (PLEG): container finished" podID="991244ae-a1ce-4c63-a5cc-efef8d29c806" containerID="241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81" exitCode=0
Sep 15 21:52:06.004116 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:06.004041    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"
Sep 15 21:52:06.004116 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:06.004088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb" event={"ID":"991244ae-a1ce-4c63-a5cc-efef8d29c806","Type":"ContainerDied","Data":"241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81"}
Sep 15 21:52:06.004228 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:06.004133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb" event={"ID":"991244ae-a1ce-4c63-a5cc-efef8d29c806","Type":"ContainerDied","Data":"d45c88a0f5dc8b98b32ce8d54e55b80c5e1b30e219856e6794ef0b2db6051e00"}
Sep 15 21:52:06.004228 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:06.004156    2563 scope.go:117] "RemoveContainer" containerID="241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81"
Sep 15 21:52:06.013422 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:06.013407    2563 scope.go:117] "RemoveContainer" containerID="241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81"
Sep 15 21:52:06.013705 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:52:06.013687    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81\": container with ID starting with 241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81 not found: ID does not exist" containerID="241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81"
Sep 15 21:52:06.013779 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:06.013711    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81"} err="failed to get container status \"241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81\": rpc error: code = NotFound desc = could not find container \"241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81\": container with ID starting with 241927e76424998108bfba1808dd9579e1b5692d07bcf0f72ad7235337c48a81 not found: ID does not exist"
Sep 15 21:52:06.026079 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:06.026058    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"]
Sep 15 21:52:06.031325 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:06.031299    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f74dbafa-kserve-864dff9f5-cv9xb"]
Sep 15 21:52:06.527192 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:06.527165    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="991244ae-a1ce-4c63-a5cc-efef8d29c806" path="/var/lib/kubelet/pods/991244ae-a1ce-4c63-a5cc-efef8d29c806/volumes"
Sep 15 21:52:30.056759 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:30.056720    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"]
Sep 15 21:52:30.057240 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:30.056970    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2" podUID="6e641850-54c7-47c8-a646-69fc7e89cff7" containerName="maas-api" containerID="cri-o://dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934" gracePeriod=30
Sep 15 21:52:31.299929 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.299899    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:52:31.419025 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.419002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-metrics-certs\") pod \"6e641850-54c7-47c8-a646-69fc7e89cff7\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") "
Sep 15 21:52:31.419212 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.419042    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k7z9\" (UniqueName: \"kubernetes.io/projected/6e641850-54c7-47c8-a646-69fc7e89cff7-kube-api-access-8k7z9\") pod \"6e641850-54c7-47c8-a646-69fc7e89cff7\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") "
Sep 15 21:52:31.419212 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.419082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e641850-54c7-47c8-a646-69fc7e89cff7-openshift-service-ca\") pod \"6e641850-54c7-47c8-a646-69fc7e89cff7\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") "
Sep 15 21:52:31.419342 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.419252    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-maas-api-tls\") pod \"6e641850-54c7-47c8-a646-69fc7e89cff7\" (UID: \"6e641850-54c7-47c8-a646-69fc7e89cff7\") "
Sep 15 21:52:31.419497 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.419473    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e641850-54c7-47c8-a646-69fc7e89cff7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e641850-54c7-47c8-a646-69fc7e89cff7" (UID: "6e641850-54c7-47c8-a646-69fc7e89cff7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:52:31.419598 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.419586    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e641850-54c7-47c8-a646-69fc7e89cff7-openshift-service-ca\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:52:31.421132 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.421103    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6e641850-54c7-47c8-a646-69fc7e89cff7" (UID: "6e641850-54c7-47c8-a646-69fc7e89cff7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:52:31.421132 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.421126    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e641850-54c7-47c8-a646-69fc7e89cff7" (UID: "6e641850-54c7-47c8-a646-69fc7e89cff7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:52:31.421257 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.421139    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e641850-54c7-47c8-a646-69fc7e89cff7-kube-api-access-8k7z9" (OuterVolumeSpecName: "kube-api-access-8k7z9") pod "6e641850-54c7-47c8-a646-69fc7e89cff7" (UID: "6e641850-54c7-47c8-a646-69fc7e89cff7"). InnerVolumeSpecName "kube-api-access-8k7z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:52:31.520245 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.520215    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k7z9\" (UniqueName: \"kubernetes.io/projected/6e641850-54c7-47c8-a646-69fc7e89cff7-kube-api-access-8k7z9\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:52:31.520245 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.520240    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-maas-api-tls\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:52:31.520245 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:31.520249    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e641850-54c7-47c8-a646-69fc7e89cff7-metrics-certs\") on node \"ip-10-0-143-191.ec2.internal\" DevicePath \"\""
Sep 15 21:52:32.147024 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:32.146993    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e641850-54c7-47c8-a646-69fc7e89cff7" containerID="dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934" exitCode=0
Sep 15 21:52:32.147195 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:32.147032    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"
Sep 15 21:52:32.147195 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:32.147060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2" event={"ID":"6e641850-54c7-47c8-a646-69fc7e89cff7","Type":"ContainerDied","Data":"dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934"}
Sep 15 21:52:32.147195 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:32.147105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2" event={"ID":"6e641850-54c7-47c8-a646-69fc7e89cff7","Type":"ContainerDied","Data":"a7d725d6da550c2fc62a8a0bec4d272d2539cac07aa74a3ad1b2fa1b824e420c"}
Sep 15 21:52:32.147195 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:32.147125    2563 scope.go:117] "RemoveContainer" containerID="dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934"
Sep 15 21:52:32.157093 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:32.157075    2563 scope.go:117] "RemoveContainer" containerID="dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934"
Sep 15 21:52:32.157445 ip-10-0-143-191 kubenswrapper[2563]: E0915 21:52:32.157421    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934\": container with ID starting with dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934 not found: ID does not exist" containerID="dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934"
Sep 15 21:52:32.157566 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:32.157450    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934"} err="failed to get container status \"dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934\": rpc error: code = NotFound desc = could not find container \"dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934\": container with ID starting with dc501ddc6ba72811f42b7843c62d490db68657adccada74d0ac1288985084934 not found: ID does not exist"
Sep 15 21:52:32.168795 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:32.168774    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"]
Sep 15 21:52:32.172340 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:32.172321    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c78f66949-2jnj2"]
Sep 15 21:52:32.526606 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:32.526545    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e641850-54c7-47c8-a646-69fc7e89cff7" path="/var/lib/kubelet/pods/6e641850-54c7-47c8-a646-69fc7e89cff7/volumes"
Sep 15 21:52:45.023603 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:45.023570    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84587fcc9d-kgk94_48b7c0e0-7f46-4e03-bce8-60c82285ac30/authorino/0.log"
Sep 15 21:52:51.344518 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:51.344487    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-568b8bf8c4-7s6pt_fdbd334c-230c-4d9b-8c8e-659d75336226/copy-manifests/0.log"
Sep 15 21:52:51.356778 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:51.356754    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-568b8bf8c4-7s6pt_fdbd334c-230c-4d9b-8c8e-659d75336226/manager/0.log"
Sep 15 21:52:52.206168 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:52.206142    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825130-9vz4j_8703060e-0e53-479a-b273-75695a095ad9/cleanup/0.log"
Sep 15 21:52:52.431300 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:52.431271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-v7kd6_bb8178b4-a139-4ffd-a84b-0434cb25075a/postgres/0.log"
Sep 15 21:52:53.225026 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:53.224995    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84587fcc9d-kgk94_48b7c0e0-7f46-4e03-bce8-60c82285ac30/authorino/0.log"
Sep 15 21:52:54.574481 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:54.574451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbb5b4bf7-8wvnx_260f8851-3890-494c-99a4-0062bbc2327c/kube-auth-proxy/0.log"
Sep 15 21:52:55.087045 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:55.087017    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-q7g9m_5aab6349-f631-40d4-8e1b-6ad8fbab3f7a/payload-processing/0.log"
Sep 15 21:52:55.673829 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:55.673803    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-svkkf_97a280e4-ee73-44ad-8d9a-e4284eaf43f3/main/0.log"
Sep 15 21:52:56.138321 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:56.138295    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-bbscv_fbc46b7c-9dce-444d-8a09-cd2d596a698e/main/0.log"
Sep 15 21:52:56.254063 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:52:56.254040    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-64dqf_dabdded8-ab58-45bc-b4e8-0c982279812e/main/0.log"
Sep 15 21:53:02.440592 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.440538    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ljsqz/must-gather-578ws"]
Sep 15 21:53:02.441242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.441085    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991244ae-a1ce-4c63-a5cc-efef8d29c806" containerName="main"
Sep 15 21:53:02.441242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.441100    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="991244ae-a1ce-4c63-a5cc-efef8d29c806" containerName="main"
Sep 15 21:53:02.441242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.441125    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e641850-54c7-47c8-a646-69fc7e89cff7" containerName="maas-api"
Sep 15 21:53:02.441242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.441130    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e641850-54c7-47c8-a646-69fc7e89cff7" containerName="maas-api"
Sep 15 21:53:02.441242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.441194    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="991244ae-a1ce-4c63-a5cc-efef8d29c806" containerName="main"
Sep 15 21:53:02.441242 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.441205    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e641850-54c7-47c8-a646-69fc7e89cff7" containerName="maas-api"
Sep 15 21:53:02.444371 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.444340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ljsqz/must-gather-578ws"
Sep 15 21:53:02.446795 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.446770    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ljsqz\"/\"kube-root-ca.crt\""
Sep 15 21:53:02.447265 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.447247    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ljsqz\"/\"default-dockercfg-gfrqc\""
Sep 15 21:53:02.447265 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.447256    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ljsqz\"/\"openshift-service-ca.crt\""
Sep 15 21:53:02.453323 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.453300    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ljsqz/must-gather-578ws"]
Sep 15 21:53:02.593636 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.593605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a5140864-b5ed-4c29-9fe2-1641739bfb41-must-gather-output\") pod \"must-gather-578ws\" (UID: \"a5140864-b5ed-4c29-9fe2-1641739bfb41\") " pod="openshift-must-gather-ljsqz/must-gather-578ws"
Sep 15 21:53:02.593636 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.593639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km2qs\" (UniqueName: \"kubernetes.io/projected/a5140864-b5ed-4c29-9fe2-1641739bfb41-kube-api-access-km2qs\") pod \"must-gather-578ws\" (UID: \"a5140864-b5ed-4c29-9fe2-1641739bfb41\") " pod="openshift-must-gather-ljsqz/must-gather-578ws"
Sep 15 21:53:02.694747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.694679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a5140864-b5ed-4c29-9fe2-1641739bfb41-must-gather-output\") pod \"must-gather-578ws\" (UID: \"a5140864-b5ed-4c29-9fe2-1641739bfb41\") " pod="openshift-must-gather-ljsqz/must-gather-578ws"
Sep 15 21:53:02.694747 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.694709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km2qs\" (UniqueName: \"kubernetes.io/projected/a5140864-b5ed-4c29-9fe2-1641739bfb41-kube-api-access-km2qs\") pod \"must-gather-578ws\" (UID: \"a5140864-b5ed-4c29-9fe2-1641739bfb41\") " pod="openshift-must-gather-ljsqz/must-gather-578ws"
Sep 15 21:53:02.695015 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.694985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a5140864-b5ed-4c29-9fe2-1641739bfb41-must-gather-output\") pod \"must-gather-578ws\" (UID: \"a5140864-b5ed-4c29-9fe2-1641739bfb41\") " pod="openshift-must-gather-ljsqz/must-gather-578ws"
Sep 15 21:53:02.702791 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.702771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km2qs\" (UniqueName: \"kubernetes.io/projected/a5140864-b5ed-4c29-9fe2-1641739bfb41-kube-api-access-km2qs\") pod \"must-gather-578ws\" (UID: \"a5140864-b5ed-4c29-9fe2-1641739bfb41\") " pod="openshift-must-gather-ljsqz/must-gather-578ws"
Sep 15 21:53:02.754313 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:02.754285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ljsqz/must-gather-578ws"
Sep 15 21:53:03.022017 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:03.021990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ljsqz/must-gather-578ws"]
Sep 15 21:53:03.023849 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:53:03.023826    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5140864_b5ed_4c29_9fe2_1641739bfb41.slice/crio-9a0fbd5b2b45eb62b07dda145e0348e1a738d893416409409921662914223133 WatchSource:0}: Error finding container 9a0fbd5b2b45eb62b07dda145e0348e1a738d893416409409921662914223133: Status 404 returned error can't find the container with id 9a0fbd5b2b45eb62b07dda145e0348e1a738d893416409409921662914223133
Sep 15 21:53:03.025774 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:03.025756    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:53:03.317921 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:03.317839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljsqz/must-gather-578ws" event={"ID":"a5140864-b5ed-4c29-9fe2-1641739bfb41","Type":"ContainerStarted","Data":"9a0fbd5b2b45eb62b07dda145e0348e1a738d893416409409921662914223133"}
Sep 15 21:53:04.327029 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:04.326980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljsqz/must-gather-578ws" event={"ID":"a5140864-b5ed-4c29-9fe2-1641739bfb41","Type":"ContainerStarted","Data":"fa819953f7154167eb77a668d683cb7e7494f64315686082f9b84080fabbc51c"}
Sep 15 21:53:04.327558 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:04.327044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljsqz/must-gather-578ws" event={"ID":"a5140864-b5ed-4c29-9fe2-1641739bfb41","Type":"ContainerStarted","Data":"ed7510ca646bf7db42d266e6f5bdb83c0f1a3664a11d449ca7ea97c01384cff9"}
Sep 15 21:53:04.342949 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:04.342900    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ljsqz/must-gather-578ws" podStartSLOduration=1.649450525 podStartE2EDuration="2.34288306s" totalImagesPullingTime="693.432535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:53:04.341394065 +0000 UTC m=+2574.304451447" watchObservedRunningTime="2026-09-15 21:53:04.34288306 +0000 UTC m=+2574.305940478"
Sep 15 21:53:05.271014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:05.270977    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2mthm_e74e7fd1-2672-4c85-b798-a3f036fb7681/global-pull-secret-syncer/0.log"
Sep 15 21:53:05.428618 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:05.428591    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b5ndc_d76ae1c9-7503-48a0-9eed-5b3539718a70/konnectivity-agent/0.log"
Sep 15 21:53:05.532228 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:05.532156    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-191.ec2.internal_bfa4cb4183b054641aa8c971a669467e/haproxy/0.log"
Sep 15 21:53:10.288686 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:10.288649    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84587fcc9d-kgk94_48b7c0e0-7f46-4e03-bce8-60c82285ac30/authorino/0.log"
Sep 15 21:53:11.953974 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:11.953947    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_806f2b57-c8d5-4011-85b8-3f61cee58e90/alertmanager/0.log"
Sep 15 21:53:11.981745 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:11.981702    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_806f2b57-c8d5-4011-85b8-3f61cee58e90/config-reloader/0.log"
Sep 15 21:53:12.005868 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.005835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_806f2b57-c8d5-4011-85b8-3f61cee58e90/kube-rbac-proxy-web/0.log"
Sep 15 21:53:12.029110 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.029081    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_806f2b57-c8d5-4011-85b8-3f61cee58e90/kube-rbac-proxy/0.log"
Sep 15 21:53:12.056589 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.056478    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_806f2b57-c8d5-4011-85b8-3f61cee58e90/kube-rbac-proxy-metric/0.log"
Sep 15 21:53:12.079957 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.079928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_806f2b57-c8d5-4011-85b8-3f61cee58e90/prom-label-proxy/0.log"
Sep 15 21:53:12.101254 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.101232    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_806f2b57-c8d5-4011-85b8-3f61cee58e90/init-config-reloader/0.log"
Sep 15 21:53:12.186308 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.186279    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-z5ft4_720e2ec7-2e6e-4807-9fe7-111093f9649a/kube-state-metrics/0.log"
Sep 15 21:53:12.205860 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.205778    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-z5ft4_720e2ec7-2e6e-4807-9fe7-111093f9649a/kube-rbac-proxy-main/0.log"
Sep 15 21:53:12.227266 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.227226    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-z5ft4_720e2ec7-2e6e-4807-9fe7-111093f9649a/kube-rbac-proxy-self/0.log"
Sep 15 21:53:12.413441 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.413410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dj85v_61172647-ba80-42bd-9ef8-3a57a9afd08e/node-exporter/0.log"
Sep 15 21:53:12.435400 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.435369    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dj85v_61172647-ba80-42bd-9ef8-3a57a9afd08e/kube-rbac-proxy/0.log"
Sep 15 21:53:12.476706 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.476640    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dj85v_61172647-ba80-42bd-9ef8-3a57a9afd08e/init-textfile/0.log"
Sep 15 21:53:12.604723 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.604579    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lmht9_412e6858-6a69-46ca-8bf0-4002bf13f748/kube-rbac-proxy-main/0.log"
Sep 15 21:53:12.643159 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.643115    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lmht9_412e6858-6a69-46ca-8bf0-4002bf13f748/kube-rbac-proxy-self/0.log"
Sep 15 21:53:12.672979 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.672953    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lmht9_412e6858-6a69-46ca-8bf0-4002bf13f748/openshift-state-metrics/0.log"
Sep 15 21:53:12.887638 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.887607    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-tlkbf_dd2f243b-db86-4a5a-b0ae-820c1e4869ee/prometheus-operator/0.log"
Sep 15 21:53:12.908555 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.908522    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-tlkbf_dd2f243b-db86-4a5a-b0ae-820c1e4869ee/kube-rbac-proxy/0.log"
Sep 15 21:53:12.937293 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.937245    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-j5649_5514f3e1-2f67-4d32-bce2-72f7de833968/prometheus-operator-admission-webhook/0.log"
Sep 15 21:53:12.970113 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.970086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7db6bd5cb5-nkb4b_7cb18cd9-b40f-457c-aecc-ac66416402c8/telemeter-client/0.log"
Sep 15 21:53:12.998656 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:12.998624    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7db6bd5cb5-nkb4b_7cb18cd9-b40f-457c-aecc-ac66416402c8/reload/0.log"
Sep 15 21:53:13.028760 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:13.028724    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7db6bd5cb5-nkb4b_7cb18cd9-b40f-457c-aecc-ac66416402c8/kube-rbac-proxy/0.log"
Sep 15 21:53:14.582633 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:14.582602    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-r9l8p_28e7256c-de77-4363-9395-2004b91d28f6/networking-console-plugin/0.log"
Sep 15 21:53:15.389858 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.389826    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"]
Sep 15 21:53:15.396369 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.396312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.400232 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.400202    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"]
Sep 15 21:53:15.441365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.440830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-proc\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.441365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.440913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-podres\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.441365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.440973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k99v\" (UniqueName: \"kubernetes.io/projected/bfbc9390-b368-41b0-bf9b-30cabdb75e24-kube-api-access-9k99v\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.441365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.441040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-lib-modules\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.441365 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.441129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-sys\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.542503 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.542461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-podres\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.542724 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.542536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k99v\" (UniqueName: \"kubernetes.io/projected/bfbc9390-b368-41b0-bf9b-30cabdb75e24-kube-api-access-9k99v\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.542724 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.542595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-lib-modules\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.542724 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.542603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-podres\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.542724 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.542702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-lib-modules\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.542724 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.542712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-sys\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.543009 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.542750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-sys\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.543009 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.542769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-proc\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.543009 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.542862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfbc9390-b368-41b0-bf9b-30cabdb75e24-proc\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.551457 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.551425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k99v\" (UniqueName: \"kubernetes.io/projected/bfbc9390-b368-41b0-bf9b-30cabdb75e24-kube-api-access-9k99v\") pod \"perf-node-gather-daemonset-fsbsw\" (UID: \"bfbc9390-b368-41b0-bf9b-30cabdb75e24\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.708787 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.708701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:15.784741 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.784661    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-587c544fd5-x9xp7_6481e213-1f4a-4933-bf91-ce500d6ae171/console/0.log"
Sep 15 21:53:15.823583 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.823550    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-lsb6g_7fda35c6-f9a2-4c7b-89d7-74a1a74b6847/download-server/0.log"
Sep 15 21:53:15.996367 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:15.996316    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"]
Sep 15 21:53:15.997815 ip-10-0-143-191 kubenswrapper[2563]: W0915 21:53:15.997790    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbfbc9390_b368_41b0_bf9b_30cabdb75e24.slice/crio-fe6ab19077d3219ae24d5c7c7bdc4f746f4076b9f80a3901253d013b3b40622d WatchSource:0}: Error finding container fe6ab19077d3219ae24d5c7c7bdc4f746f4076b9f80a3901253d013b3b40622d: Status 404 returned error can't find the container with id fe6ab19077d3219ae24d5c7c7bdc4f746f4076b9f80a3901253d013b3b40622d
Sep 15 21:53:16.434943 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:16.434720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw" event={"ID":"bfbc9390-b368-41b0-bf9b-30cabdb75e24","Type":"ContainerStarted","Data":"5a02c4dbdbfccadae3f8fdcbf0043762b6e2d781dc0d3663a5bcefcdb6db9477"}
Sep 15 21:53:16.434943 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:16.434790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw" event={"ID":"bfbc9390-b368-41b0-bf9b-30cabdb75e24","Type":"ContainerStarted","Data":"fe6ab19077d3219ae24d5c7c7bdc4f746f4076b9f80a3901253d013b3b40622d"}
Sep 15 21:53:16.435198 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:16.434953    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:16.454147 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:16.454098    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw" podStartSLOduration=1.454082749 podStartE2EDuration="1.454082749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:53:16.45005159 +0000 UTC m=+2586.413108972" watchObservedRunningTime="2026-09-15 21:53:16.454082749 +0000 UTC m=+2586.417140129"
Sep 15 21:53:17.536106 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:17.536080    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m9jvn_04d67dbc-688d-42da-a42a-66aa0b7258e8/dns/0.log"
Sep 15 21:53:17.560182 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:17.560142    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m9jvn_04d67dbc-688d-42da-a42a-66aa0b7258e8/kube-rbac-proxy/0.log"
Sep 15 21:53:17.744433 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:17.744403    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sds6c_8060115d-0d03-495c-bd01-c0a4315c501f/dns-node-resolver/0.log"
Sep 15 21:53:18.385743 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:18.385710    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hcnhm_61e8e166-9aa3-4f53-b9d2-564ce990235c/node-ca/0.log"
Sep 15 21:53:20.197771 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:20.197740    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbb5b4bf7-8wvnx_260f8851-3890-494c-99a4-0062bbc2327c/kube-auth-proxy/0.log"
Sep 15 21:53:20.334856 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:20.334829    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-q7g9m_5aab6349-f631-40d4-8e1b-6ad8fbab3f7a/payload-processing/0.log"
Sep 15 21:53:21.236835 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:21.236790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2gb66_691cfa1a-d482-4ccb-91d4-e0dc621dc229/serve-healthcheck-canary/0.log"
Sep 15 21:53:21.754014 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:21.753986    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-wzf6l_fd1f9f59-883a-4b1c-80af-41e5ebc914f0/insights-operator/0.log"
Sep 15 21:53:21.756374 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:21.756335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-wzf6l_fd1f9f59-883a-4b1c-80af-41e5ebc914f0/insights-operator/1.log"
Sep 15 21:53:21.922619 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:21.922589    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cv55t_eb1e5c25-047e-44b4-b262-c21e8f1a2f3c/kube-rbac-proxy/0.log"
Sep 15 21:53:21.949827 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:21.949799    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cv55t_eb1e5c25-047e-44b4-b262-c21e8f1a2f3c/exporter/0.log"
Sep 15 21:53:21.975133 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:21.975106    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cv55t_eb1e5c25-047e-44b4-b262-c21e8f1a2f3c/extractor/0.log"
Sep 15 21:53:22.453182 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:22.453147    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-fsbsw"
Sep 15 21:53:25.026214 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:25.026181    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-568b8bf8c4-7s6pt_fdbd334c-230c-4d9b-8c8e-659d75336226/manager/0.log"
Sep 15 21:53:25.045459 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:25.045436    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-568b8bf8c4-7s6pt_fdbd334c-230c-4d9b-8c8e-659d75336226/copy-manifests/0.log"
Sep 15 21:53:25.873178 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:25.873148    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbb5b4bf7-8wvnx_260f8851-3890-494c-99a4-0062bbc2327c/kube-auth-proxy/0.log"
Sep 15 21:53:25.970995 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:25.970962    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-q7g9m_5aab6349-f631-40d4-8e1b-6ad8fbab3f7a/payload-processing/0.log"
Sep 15 21:53:26.439559 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:26.439525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-ffdbb6cbd-h7f66_b86bb871-3883-4b9c-b98c-bafa5963bdd6/manager/0.log"
Sep 15 21:53:26.494188 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:26.494153    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-7bjv8_589b7486-5e05-42d2-8e2b-98a33276d7f0/openshift-lws-operator/0.log"
Sep 15 21:53:28.341910 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:28.341873    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbb5b4bf7-8wvnx_260f8851-3890-494c-99a4-0062bbc2327c/kube-auth-proxy/0.log"
Sep 15 21:53:28.442983 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:28.442956    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-q7g9m_5aab6349-f631-40d4-8e1b-6ad8fbab3f7a/payload-processing/0.log"
Sep 15 21:53:31.425619 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:31.425589    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-zbsst_abd57f26-b082-4f2f-af4e-b444ff67a234/migrator/0.log"
Sep 15 21:53:31.446780 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:31.446758    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-zbsst_abd57f26-b082-4f2f-af4e-b444ff67a234/graceful-termination/0.log"
Sep 15 21:53:31.827566 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:31.827527    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-q6lc7_5e7474e7-64cf-464e-b7f4-d8114506b756/kube-storage-version-migrator-operator/1.log"
Sep 15 21:53:31.828415 ip-10-0-143-191 kubenswrapper[2563]: I0915 21:53:31.828385    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-q6lc7_5e7474e7-64cf-464e-b7f4-d8114506b756/kube-storage-version-migrator-operator/0.log"