Sep 17 01:20:12.768163 ip-10-0-130-87 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 01:20:12.768176 ip-10-0-130-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 01:20:12.768185 ip-10-0-130-87 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 01:20:12.768381 ip-10-0-130-87 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 01:20:22.997660 ip-10-0-130-87 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 01:20:22.997681 ip-10-0-130-87 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 70062f78164a4268b3a9eee66e8ad942 --
Sep 17 01:22:01.227853 ip-10-0-130-87 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 01:22:01.621918 ip-10-0-130-87 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 01:22:01.621918 ip-10-0-130-87 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 01:22:01.621918 ip-10-0-130-87 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 01:22:01.621918 ip-10-0-130-87 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 01:22:01.621918 ip-10-0-130-87 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 01:22:01.624823 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.624735    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 01:22:01.627971 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627957    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 01:22:01.627971 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627971    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627975    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627978    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627981    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627984    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627987    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627990    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627992    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627995    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.627997    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628000    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628002    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628005    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628007    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628010    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628012    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628015    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628033    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628036    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 01:22:01.628043 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628041    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628045    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628050    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628054    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628057    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628060    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628063    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628065    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628068    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628070    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628073    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628076    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628078    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628081    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628083    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628086    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628089    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628091    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628094    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 01:22:01.628500 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628096    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628099    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628101    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628103    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628106    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628109    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628111    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628114    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628118    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628120    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628123    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628125    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628128    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628130    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628133    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628145    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628148    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628152    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628154    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628157    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 01:22:01.628975 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628160    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628163    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628165    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628168    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628170    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628173    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628175    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628178    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628180    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628183    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628185    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628188    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628191    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628193    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628196    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628199    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628201    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628204    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628206    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628209    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628211    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 01:22:01.629488 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628217    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628220    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628222    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628225    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628227    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628229    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628233    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628236    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628239    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628241    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628244    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628246    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628249    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628251    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628254    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628256    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628258    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628262    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.628266    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 01:22:01.630008 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629756    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629764    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629768    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629771    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629774    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629777    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629780    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629783    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629786    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629789    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629791    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629794    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629797    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629800    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629804    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629806    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629809    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629811    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629814    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629817    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 01:22:01.630548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629821    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629823    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629827    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629831    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629835    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629837    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629840    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629843    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629846    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629848    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629851    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629854    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629857    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629859    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629862    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629864    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629867    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629870    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629872    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629874    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 01:22:01.631053 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629877    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629880    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629883    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629885    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629888    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629891    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629893    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629896    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629899    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629901    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629904    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629908    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629911    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629913    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629916    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629918    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629921    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629923    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629926    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 01:22:01.631564 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629928    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629931    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629934    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629936    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629938    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629941    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629946    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629949    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629951    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629954    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629956    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629959    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629962    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629966    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629969    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629972    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629975    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629977    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629980    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629982    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 01:22:01.632036 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629984    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629987    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629990    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629992    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.629995    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630000    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630004    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630006    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630009    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630012    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630015    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630018    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630020    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630023    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630026    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630028    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630030    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630033    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630036    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 01:22:01.632551 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630039    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630394    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630399    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630402    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630404    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630407    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630409    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630412    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630414    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630417    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630419    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630422    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630424    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630427    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630430    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630432    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630435    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630438    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630441    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630444    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 01:22:01.633014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630446    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630449    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630451    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630454    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630456    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630459    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630461    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630465    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630468    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630471    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630474    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630477    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630480    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630483    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630485    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630488    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630491    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630493    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630496    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 01:22:01.633505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630498    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630500    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630503    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630505    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630508    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630510    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630513    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630515    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630518    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630521    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630524    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630528    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630545    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630549    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630552    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630555    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630558    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630560    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630563    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630565    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 01:22:01.633988 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630568    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630571    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630573    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630576    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630578    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630581    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630584    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630587    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630589    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630592    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630595    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630597    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630600    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630602    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630605    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630607    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630610    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630612    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630615    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630617    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 01:22:01.634505 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630620    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630623    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630626    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630628    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630632    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630635    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630637    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630639    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630642    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630645    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630648    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630651    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630654    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630656    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630659    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630661    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630664    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630666    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630668    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 01:22:01.635000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630672    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.630675    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630745    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630783    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630789    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630793    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630798    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630801    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630810    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630814    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630817    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630820    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630825    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630829    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630832    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630836    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630840    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630843    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630846    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630849    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630853    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630856    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630859    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630862    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 01:22:01.635461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630866    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630869    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630872    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630875    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630878    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630881    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630884    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630887    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630891    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630893    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630897    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630900    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630903    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630906    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630909    2578 flags.go:64] FLAG: --enable-server="true"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630912    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630916    2578 flags.go:64] FLAG: --event-burst="100"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630919    2578 flags.go:64] FLAG: --event-qps="50"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630921    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630924    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630928    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630932    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630935    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630939    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630942    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 17 01:22:01.636050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630945    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630948    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630951    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630954    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630956    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630959    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630962    2578 flags.go:64] FLAG: --feature-gates=""
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630966    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630969    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630972    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630975    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630978    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630981    2578 flags.go:64] FLAG: --help="false"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630984    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-130-87.ec2.internal"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630987    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630990    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630993    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630996    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.630999    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631002    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631005    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631007    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631010    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 01:22:01.636657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631013    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631017    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631019    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631022    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631026    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631029    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631032    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631036    2578 flags.go:64] FLAG: --lock-file=""
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631039    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631042    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631045    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631050    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631054    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631057    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631060    2578 flags.go:64] FLAG: --logging-format="text"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631063    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631066    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631069    2578 flags.go:64] FLAG: --manifest-url=""
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631072    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631076    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631079    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631083    2578 flags.go:64] FLAG: --max-pods="110"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631086    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631088    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631091    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 01:22:01.637245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631094    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631097    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631100    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631103    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631109    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631112    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631115    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631119    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631122    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631127    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631129    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631134    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631137    2578 flags.go:64] FLAG: --port="10250"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631140    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631143    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0de9cc718d7f03aa4"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631147    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631150    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631154    2578 flags.go:64] FLAG: --register-node="true"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631156    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631160    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631163    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631166    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631168    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631172    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631175    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 01:22:01.637941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631178    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631181    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631183    2578 flags.go:64] FLAG: --runonce="false"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631186    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631189    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631192    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631195    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631198    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631201    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631204    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631208    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631210    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631213    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631216    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631219    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631222    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631225    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631227    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631234    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631237    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631240    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631244    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631248    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631251    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631254    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 01:22:01.638647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631258    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631261    2578 flags.go:64] FLAG: --v="2"
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631264    2578 flags.go:64] FLAG: --version="false"
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631268    2578 flags.go:64] FLAG: --vmodule=""
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631272    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.631275    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631385    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631389    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631392    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631395    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631398    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631400    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631403    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631405    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631408    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631410    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631413    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631416    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631418    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631421    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631424    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 01:22:01.639707 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631426    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631429    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631431    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631434    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631438    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631440    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631443    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631445    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631450    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631452    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631455    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631458    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631460    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631463    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631466    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631468    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631471    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631474    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631477    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631479    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 01:22:01.640566 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631482    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631485    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631487    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631490    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631493    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631495    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631498    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631501    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631503    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631506    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631509    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631511    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631514    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631517    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631521    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631524    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631529    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631546    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631550    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 01:22:01.641116 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631553    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631558    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631561    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631563    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631566    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631569    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631572    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631575    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631578    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631581    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631583    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631587    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631590    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631593    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631596    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631599    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631602    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631604    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631607    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 01:22:01.641615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631610    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631613    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631616    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631618    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631621    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631623    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631633    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631636    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631639    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631641    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631646    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631648    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631651    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631654    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631657    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631660    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631663    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631666    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631668    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631672    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 01:22:01.642081 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631676    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631680    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631683    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631686    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631689    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.631691    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.632382    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.641398    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.641413    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.641433    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.641439    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641491    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641498    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641501    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 01:22:01.642615 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641504    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641507    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641511    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641514    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641517    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641520    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641522    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641526    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641530    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641553    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641557    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641559    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641562    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641565    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641568    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641571    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641574    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641577    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641580    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 01:22:01.642969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641582    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641585    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641588    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641590    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641593    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641597    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641600    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641605    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641608    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641611    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641614    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641617    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641620    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641623    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641626    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641629    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641632    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641635    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641638    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641641    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 01:22:01.643427 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641643    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641646    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641649    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641651    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641654    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641656    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641659    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641661    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641664    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641667    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641669    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641672    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641675    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641677    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641680    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641683    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641687    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641690    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641694    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 01:22:01.643926 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641697    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641700    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641703    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641705    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641708    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641711    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641713    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641716    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641719    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641721    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641724    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641727    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641730    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641732    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641735    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641737    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641740    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641743    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641746    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641749    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 01:22:01.644406 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641751    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641754    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641757    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641759    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641762    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641765    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641767    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641770    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641772    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641775    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641778    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641781    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641783    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641786    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641789    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641792    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641795    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 01:22:01.644969 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641797    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.641802    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641926    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641931    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641934    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641937    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641940    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641943    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641946    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641949    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641952    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641955    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641958    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641961    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641965    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 01:22:01.645387 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641968    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641971    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641974    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641976    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641979    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641982    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641984    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641987    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641989    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641992    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641995    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.641998    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642001    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642003    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642006    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642009    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642011    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642014    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642016    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642019    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 01:22:01.645783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642021    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642025    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642029    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642031    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642034    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642037    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642039    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642042    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642044    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642047    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642050    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642052    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642056    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642058    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642061    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642063    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642066    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642069    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642071    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 01:22:01.646266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642075    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642078    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642081    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642084    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642087    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642091    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642094    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642097    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642099    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642102    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642105    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642108    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642110    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642113    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642116    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642118    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642121    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642124    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642126    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642129    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 01:22:01.646722 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642131    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642134    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642136    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642139    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642142    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642144    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642148    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642150    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642153    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642155    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642158    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642160    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642163    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642166    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642168    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642171    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642173    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642176    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642179    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642181    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 01:22:01.647208 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642184    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642186    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642190    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642194    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642196    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642199    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:01.642202    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.642207    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.642314    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.646250    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.647274    2578 server.go:1019] "Starting client certificate rotation"
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.647368    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 01:22:01.647697 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.647399    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 01:22:01.650396 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.650382    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 01:22:01.651621 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.651573    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 01:22:01.660056 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.660039    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 17 01:22:01.663327 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.663307    2578 log.go:25] "Validated CRI v1 image API"
Sep 17 01:22:01.664963 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.664947    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 01:22:01.670419 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.670390    2578 fs.go:135] Filesystem UUIDs: map[27010eb0-979b-48af-aa6c-99b015aa16a1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e6a09091-ff58-440a-b6dc-b2e6afa16f77:/dev/nvme0n1p4]
Sep 17 01:22:01.670484 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.670419    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 01:22:01.676306 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.676183    2578 manager.go:216] Machine: {Timestamp:2026-09-17 01:22:01.673883425 +0000 UTC m=+0.344399505 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103709 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fe7e68d56af2e9abce4e2a20456a8 SystemUUID:ec2fe7e6-8d56-af2e-9abc-e4e2a20456a8 BootID:70062f78-164a-4268-b3a9-eee66e8ad942 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:65:f0:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:65:f0:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:4c:43:1f:ee:de Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 01:22:01.676306 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.676291    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 01:22:01.676422 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.676396    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 01:22:01.679883 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.679861    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 01:22:01.680960 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.679886    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-87.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 01:22:01.681141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.681117    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 01:22:01.681735 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.681712    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 01:22:01.681818 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.681739    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 01:22:01.681818 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.681752    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 01:22:01.682455 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.682443    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 01:22:01.683777 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.683764    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 01:22:01.688092 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.688081    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 01:22:01.688213 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.688203    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 01:22:01.688247 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.688233    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 01:22:01.688247 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.688243    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 01:22:01.688311 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.688255    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 01:22:01.688311 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.688267    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 17 01:22:01.688311 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.688279    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 01:22:01.688661 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.688649    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 01:22:01.688717 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.688672    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 01:22:01.691728 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.691711    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 01:22:01.694693 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.694679    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 01:22:01.694758 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.694708    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 01:22:01.696718 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696706    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 01:22:01.696758 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696724    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 01:22:01.696758 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696730    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 01:22:01.696758 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696736    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 01:22:01.696758 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696744    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 01:22:01.696758 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696749    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 01:22:01.696893 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696762    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 01:22:01.696893 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696767    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 01:22:01.696893 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696774    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 01:22:01.696893 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696779    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 01:22:01.696893 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696787    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 01:22:01.696893 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.696793    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 01:22:01.697731 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.697722    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 01:22:01.697731 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.697731    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 01:22:01.698315 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.698291    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 01:22:01.698347 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.698297    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 01:22:01.699666 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.699652    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-87.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 01:22:01.701630 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.701613    2578 server.go:1268] "Started kubelet"
Sep 17 01:22:01.701725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.701703    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 01:22:01.701778 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.701687    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 01:22:01.701778 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.701765    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 01:22:01.702298 ip-10-0-130-87 systemd[1]: Started Kubernetes Kubelet.
Sep 17 01:22:01.702976 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.702954    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tqjrm"
Sep 17 01:22:01.703694 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.703679    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 01:22:01.704475 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.704457    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 01:22:01.707311 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.707289    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 01:22:01.710317 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.710299    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tqjrm"
Sep 17 01:22:01.712040 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.712024    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 01:22:01.712124 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.712027    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 01:22:01.712124 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.712116    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:01.712857 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.712837    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 01:22:01.712857 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.712837    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 01:22:01.713079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.712862    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 01:22:01.713079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.712936    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 01:22:01.713079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.712945    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 01:22:01.713079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.712949    2578 factory.go:55] Registering systemd factory
Sep 17 01:22:01.713079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.712965    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 17 01:22:01.713267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.713180    2578 factory.go:153] Registering CRI-O factory
Sep 17 01:22:01.713267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.713194    2578 factory.go:223] Registration of the crio container factory successfully
Sep 17 01:22:01.713267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.713247    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 01:22:01.713267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.713265    2578 factory.go:103] Registering Raw factory
Sep 17 01:22:01.713433 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.713283    2578 manager.go:1196] Started watching for new ooms in manager
Sep 17 01:22:01.714151 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.714135    2578 manager.go:349] Starting recovery of all containers
Sep 17 01:22:01.717560 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.717523    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 01:22:01.723303 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.723281    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-87.ec2.internal\" not found" node="ip-10-0-130-87.ec2.internal"
Sep 17 01:22:01.727609 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.727594    2578 manager.go:354] Recovery completed
Sep 17 01:22:01.729255 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.729233    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 17 01:22:01.733824 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.733811    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 01:22:01.735919 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.735904    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 01:22:01.735981 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.735933    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 01:22:01.735981 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.735948    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientPID"
Sep 17 01:22:01.736453 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.736441    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 01:22:01.736453 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.736451    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 01:22:01.736585 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.736460    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 01:22:01.736585 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.736468    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 01:22:01.738519 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.738507    2578 policy_none.go:49] "None policy: Start"
Sep 17 01:22:01.738601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.738522    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 01:22:01.738601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.738544    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 01:22:01.739099 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.739083    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 01:22:01.739164 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.739112    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 01:22:01.739164 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.739125    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientPID"
Sep 17 01:22:01.740594 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.740575    2578 policy_none.go:47] "Start"
Sep 17 01:22:01.770367 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.770223    2578 manager.go:337] "Starting Device Plugin manager"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.770411    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.770424    2578 server.go:87] "Starting device plugin registration server"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.770623    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.770636    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.770849    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.770927    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.770935    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.771290    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.771334    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.871223    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.872133    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.872172    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.872183    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientPID"
Sep 17 01:22:01.878097 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.872210    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-87.ec2.internal"
Sep 17 01:22:01.880320 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.880307    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-87.ec2.internal"
Sep 17 01:22:01.880386 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.880326    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-87.ec2.internal\": node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:01.898620 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.898600    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:01.912587 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.912553    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 01:22:01.913768 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.913747    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 01:22:01.913851 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.913801    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 01:22:01.913851 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.913826    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 01:22:01.913951 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.913912    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 01:22:01.916304 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:01.916289    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 01:22:01.999380 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:01.999352    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:02.015217 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.015199    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal"]
Sep 17 01:22:02.015273 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.015255    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 01:22:02.016163 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.016144    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 01:22:02.016232 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.016172    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 01:22:02.016232 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.016181    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientPID"
Sep 17 01:22:02.018451 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.018438    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 01:22:02.018594 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.018567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.018653 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.018600    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 01:22:02.019145 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.019131    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 01:22:02.019206 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.019149    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 01:22:02.019206 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.019158    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 01:22:02.019206 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.019169    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 01:22:02.019206 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.019172    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientPID"
Sep 17 01:22:02.019206 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.019184    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientPID"
Sep 17 01:22:02.021345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.021331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.021430 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.021355    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 01:22:02.022060 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.022047    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 01:22:02.022150 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.022074    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 01:22:02.022150 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.022090    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeHasSufficientPID"
Sep 17 01:22:02.045079 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.045056    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-87.ec2.internal\" not found" node="ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.049381 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.049366    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-87.ec2.internal\" not found" node="ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.099927 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.099905    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:02.115283 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.115263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e047cec48a9cc0053170912f91d4d94a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-87.ec2.internal\" (UID: \"e047cec48a9cc0053170912f91d4d94a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.115345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.115288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cff2e4f35896379a5eb0549eb6d3cc94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal\" (UID: \"cff2e4f35896379a5eb0549eb6d3cc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.115345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.115305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cff2e4f35896379a5eb0549eb6d3cc94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal\" (UID: \"cff2e4f35896379a5eb0549eb6d3cc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.200578 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.200551    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:02.216135 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.216113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cff2e4f35896379a5eb0549eb6d3cc94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal\" (UID: \"cff2e4f35896379a5eb0549eb6d3cc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.216202 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.216082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cff2e4f35896379a5eb0549eb6d3cc94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal\" (UID: \"cff2e4f35896379a5eb0549eb6d3cc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.216245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.216206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e047cec48a9cc0053170912f91d4d94a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-87.ec2.internal\" (UID: \"e047cec48a9cc0053170912f91d4d94a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.216245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.216233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cff2e4f35896379a5eb0549eb6d3cc94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal\" (UID: \"cff2e4f35896379a5eb0549eb6d3cc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.216324 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.216286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cff2e4f35896379a5eb0549eb6d3cc94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal\" (UID: \"cff2e4f35896379a5eb0549eb6d3cc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.216324 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.216296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e047cec48a9cc0053170912f91d4d94a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-87.ec2.internal\" (UID: \"e047cec48a9cc0053170912f91d4d94a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.302278 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.302226    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:02.349224 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.349198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.352460 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.352444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:02.402901 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.402875    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:02.509514 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.509451    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:02.612168 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.612132    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:02.647272 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.647247    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 01:22:02.647812 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.647385    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 01:22:02.647812 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.647419    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 01:22:02.712880 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.712852    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:02.713361 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.713333    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 01:17:02 +0000 UTC" deadline="2028-05-27 03:53:57.062707544 +0000 UTC"
Sep 17 01:22:02.713400 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:02.713364    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14834h31m54.349348229s"
Sep 17 01:22:02.813851 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.813779    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:02.914806 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:02.914758    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:03.015753 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.015723    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:03.070804 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:03.070734    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode047cec48a9cc0053170912f91d4d94a.slice/crio-1bcacf2335fa6e319176c11917a1a0595b8e3d01266b7f2526b42367ef4f60e5 WatchSource:0}: Error finding container 1bcacf2335fa6e319176c11917a1a0595b8e3d01266b7f2526b42367ef4f60e5: Status 404 returned error can't find the container with id 1bcacf2335fa6e319176c11917a1a0595b8e3d01266b7f2526b42367ef4f60e5
Sep 17 01:22:03.071355 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:03.071334    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff2e4f35896379a5eb0549eb6d3cc94.slice/crio-eef4d18f56589de7cdac7e21810fb65d3621255801e1a44c9fdcd9888badce64 WatchSource:0}: Error finding container eef4d18f56589de7cdac7e21810fb65d3621255801e1a44c9fdcd9888badce64: Status 404 returned error can't find the container with id eef4d18f56589de7cdac7e21810fb65d3621255801e1a44c9fdcd9888badce64
Sep 17 01:22:03.077225 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.077205    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 01:22:03.116705 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.116680    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-87.ec2.internal\" not found"
Sep 17 01:22:03.195224 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.195197    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 01:22:03.213238 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.213214    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:03.228501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.228474    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 01:22:03.229606 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.229589    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 01:22:03.230628 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.230603    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal"
Sep 17 01:22:03.245432 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.245411    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 01:22:03.691071 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.690971    2578 apiserver.go:52] "Watching apiserver"
Sep 17 01:22:03.699285 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.699254    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 01:22:03.700111 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.699900    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-jsnp2","openshift-network-operator/iptables-alerter-92748","openshift-ovn-kubernetes/ovnkube-node-t8kjk","kube-system/global-pull-secret-syncer-mws4d","kube-system/konnectivity-agent-r9hm4","kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw","openshift-dns/node-resolver-mxzkv","openshift-image-registry/node-ca-22cxn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal","openshift-multus/multus-additional-cni-plugins-hvtgt","openshift-cluster-node-tuning-operator/tuned-c6xsj","openshift-multus/multus-zdnlz","openshift-multus/network-metrics-daemon-blh4d"]
Sep 17 01:22:03.702700 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.702672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.705179 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.705047    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 01:22:03.705296 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.705183    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 01:22:03.705296 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.705280    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-27bw4\""
Sep 17 01:22:03.705425 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.705398    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 01:22:03.705518 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.705503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.708674 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.707792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.708674 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.707952    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 01:22:03.708674 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.708172    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 01:22:03.708674 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.708399    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 01:22:03.708674 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.708487    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 01:22:03.708674 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.708526    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ctm22\""
Sep 17 01:22:03.710186 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.710166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:03.710289 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.710238    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:03.710521 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.710502    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 01:22:03.711962 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.711207    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 01:22:03.711962 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.711229    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 01:22:03.711962 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.711489    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 01:22:03.711962 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.711571    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 01:22:03.711962 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.711735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6bw4x\""
Sep 17 01:22:03.711962 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.711810    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 01:22:03.712968 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.712946    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 01:22:03.713974 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.713584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:03.716117 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.716095    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 01:22:03.716217 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.716156    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gjp8b\""
Sep 17 01:22:03.716282 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.716260    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 01:22:03.717238 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.717220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:03.717333 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.717287    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:03.719840 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.719821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:03.722682 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.722661    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:03.722792 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.722731    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:03.723939 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.723733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-node-log\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.723939 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.723771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4f4v\" (UniqueName: \"kubernetes.io/projected/73d5e141-cf74-4545-91ef-0e34b3c8b849-kube-api-access-m4f4v\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.723939 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.723801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-socket-dir-parent\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.723939 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.723828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvpj\" (UniqueName: \"kubernetes.io/projected/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-kube-api-access-pzvpj\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.723939 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.723855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/32d800a9-78b7-4d36-9616-e8f99b42df1f-ovn-node-metrics-cert\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.723939 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.723881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-etc-openvswitch\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.723939 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.723907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-log-socket\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.723939 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.723930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-cni-bin\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.723953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a2a1999-931a-46a1-83b0-57b1c9ca0dda-konnectivity-ca\") pod \"konnectivity-agent-r9hm4\" (UID: \"1a2a1999-931a-46a1-83b0-57b1c9ca0dda\") " pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-socket-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-sys-fs\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-cnibin\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-var-lib-cni-bin\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-hostroot\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-slash\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-run-netns\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/32d800a9-78b7-4d36-9616-e8f99b42df1f-ovnkube-config\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-registration-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-system-cni-dir\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-cni-dir\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-daemon-config\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-systemd-units\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.724481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-kubelet-config\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a2a1999-931a-46a1-83b0-57b1c9ca0dda-agent-certs\") pod \"konnectivity-agent-r9hm4\" (UID: \"1a2a1999-931a-46a1-83b0-57b1c9ca0dda\") " pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-device-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-cni-binary-copy\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-var-lib-cni-multus\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-var-lib-openvswitch\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-run-openvswitch\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-run-ovn-kubernetes\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-cni-netd\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/32d800a9-78b7-4d36-9616-e8f99b42df1f-ovnkube-script-lib\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-run-netns\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/32d800a9-78b7-4d36-9616-e8f99b42df1f-env-overrides\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724848    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt9q6\" (UniqueName: \"kubernetes.io/projected/32d800a9-78b7-4d36-9616-e8f99b42df1f-kube-api-access-mt9q6\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-dbus\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:03.725350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-run-systemd\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-run-k8s-cni-cncf-io\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.724970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-etc-kubernetes\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.725006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-kubelet\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.725038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-etc-selinux\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.725064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-var-lib-kubelet\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.725141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-conf-dir\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.725175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-run-multus-certs\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.725217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-run-ovn\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.725432    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.726148    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gmv4r\""
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.726159    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 01:22:03.726341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.726185    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 01:22:03.727327 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.727308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:03.727521 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.727501    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 01:22:03.728012 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.727992    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 01:22:03.728103 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.728012    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 01:22:03.729206 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.729186    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nwrkj\""
Sep 17 01:22:03.730354 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.730284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.732697 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.732673    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 01:22:03.732849 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.732801    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 01:22:03.732921 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.732673    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 01:22:03.734919 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.734897    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4mqqq\""
Sep 17 01:22:03.735220 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.735200    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 01:22:03.735429 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.735410    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 01:22:03.735501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.735409    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zx65r\""
Sep 17 01:22:03.735924 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.735903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.738267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.738245    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 01:22:03.738494 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.738480    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jrl7k\""
Sep 17 01:22:03.738682 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.738668    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 01:22:03.749357 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.749333    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z6z2t"
Sep 17 01:22:03.757133 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.756766    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z6z2t"
Sep 17 01:22:03.815077 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.815036    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 01:22:03.825627 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.825751 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/32d800a9-78b7-4d36-9616-e8f99b42df1f-env-overrides\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.825751 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-etc-kubernetes\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.825751 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-kubelet\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.825751 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-sysctl-d\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.825955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.825955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-etc-kubernetes\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.825955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-kubelet\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.825955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nw5q\" (UniqueName: \"kubernetes.io/projected/a07550c1-c7c1-4168-82fe-0e00fe3221dd-kube-api-access-7nw5q\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.825955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-var-lib-kubelet\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.825955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-var-lib-kubelet\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.825955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-run-multus-certs\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.825955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-run-multus-certs\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.825968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4f4v\" (UniqueName: \"kubernetes.io/projected/73d5e141-cf74-4545-91ef-0e34b3c8b849-kube-api-access-m4f4v\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c5faf65-3a14-47b8-8077-af57e33f8dfc-system-cni-dir\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/32d800a9-78b7-4d36-9616-e8f99b42df1f-ovn-node-metrics-cert\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-cni-bin\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-sys-fs\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-var-lib-kubelet\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/32d800a9-78b7-4d36-9616-e8f99b42df1f-env-overrides\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-cnibin\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-var-lib-cni-bin\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-var-lib-cni-bin\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-cni-bin\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-cnibin\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.826402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-hostroot\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-run-netns\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33989fb0-89ab-47a8-90a4-2fbd2332d884-serviceca\") pod \"node-ca-22cxn\" (UID: \"33989fb0-89ab-47a8-90a4-2fbd2332d884\") " pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-sysconfig\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-kubernetes\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-sys\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-system-cni-dir\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-systemd-units\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a2a1999-931a-46a1-83b0-57b1c9ca0dda-agent-certs\") pod \"konnectivity-agent-r9hm4\" (UID: \"1a2a1999-931a-46a1-83b0-57b1c9ca0dda\") " pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-device-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-etc-selinux\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-486xp\" (UniqueName: \"kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp\") pod \"network-check-target-jsnp2\" (UID: \"1026d5e4-b4fd-4730-b074-7973a88b774c\") " pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-cni-binary-copy\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-var-lib-cni-multus\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-var-lib-openvswitch\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-run-openvswitch\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.827079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-run-ovn-kubernetes\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/32d800a9-78b7-4d36-9616-e8f99b42df1f-ovnkube-script-lib\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6d0c858-c6bd-4ff3-a60e-10932174fd12-host-slash\") pod \"iptables-alerter-92748\" (UID: \"a6d0c858-c6bd-4ff3-a60e-10932174fd12\") " pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt9q6\" (UniqueName: \"kubernetes.io/projected/32d800a9-78b7-4d36-9616-e8f99b42df1f-kube-api-access-mt9q6\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826882    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-sys-fs\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-dbus\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.826883    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-device-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-etc-selinux\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frh2m\" (UniqueName: \"kubernetes.io/projected/33989fb0-89ab-47a8-90a4-2fbd2332d884-kube-api-access-frh2m\") pod \"node-ca-22cxn\" (UID: \"33989fb0-89ab-47a8-90a4-2fbd2332d884\") " pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ptd\" (UniqueName: \"kubernetes.io/projected/a6d0c858-c6bd-4ff3-a60e-10932174fd12-kube-api-access-l7ptd\") pod \"iptables-alerter-92748\" (UID: \"a6d0c858-c6bd-4ff3-a60e-10932174fd12\") " pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpgm\" (UniqueName: \"kubernetes.io/projected/d2d5b623-d9d2-4621-bc65-66241a002022-kube-api-access-dkpgm\") pod \"node-resolver-mxzkv\" (UID: \"d2d5b623-d9d2-4621-bc65-66241a002022\") " pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-sysctl-conf\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-run-systemd\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-run-k8s-cni-cncf-io\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d5b623-d9d2-4621-bc65-66241a002022-tmp-dir\") pod \"node-resolver-mxzkv\" (UID: \"d2d5b623-d9d2-4621-bc65-66241a002022\") " pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:03.827943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-modprobe-d\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-run-ovn-kubernetes\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-hostroot\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-systemd\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-conf-dir\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-run-ovn\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-systemd-units\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-node-log\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2d5b623-d9d2-4621-bc65-66241a002022-hosts-file\") pod \"node-resolver-mxzkv\" (UID: \"d2d5b623-d9d2-4621-bc65-66241a002022\") " pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c5faf65-3a14-47b8-8077-af57e33f8dfc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.827519    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cfhj\" (UniqueName: \"kubernetes.io/projected/7c5faf65-3a14-47b8-8077-af57e33f8dfc-kube-api-access-9cfhj\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-tuned\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.827587    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret podName:a9850b1d-04e3-4bbd-9072-a9ac2dab518a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:04.327565783 +0000 UTC m=+2.998081851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret") pod "global-pull-secret-syncer-mws4d" (UID: "a9850b1d-04e3-4bbd-9072-a9ac2dab518a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-conf-dir\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827634    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-run-netns\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-system-cni-dir\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.828898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-run-systemd\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-run-openvswitch\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-var-lib-cni-multus\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-run-k8s-cni-cncf-io\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.827968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-var-lib-openvswitch\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-dbus\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-cni-binary-copy\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/32d800a9-78b7-4d36-9616-e8f99b42df1f-ovnkube-script-lib\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-socket-dir-parent\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-socket-dir-parent\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvpj\" (UniqueName: \"kubernetes.io/projected/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-kube-api-access-pzvpj\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-node-log\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c5faf65-3a14-47b8-8077-af57e33f8dfc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07550c1-c7c1-4168-82fe-0e00fe3221dd-tmp\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-run-ovn\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-etc-openvswitch\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-etc-openvswitch\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-log-socket\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.829767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-log-socket\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a2a1999-931a-46a1-83b0-57b1c9ca0dda-konnectivity-ca\") pod \"konnectivity-agent-r9hm4\" (UID: \"1a2a1999-931a-46a1-83b0-57b1c9ca0dda\") " pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-socket-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a6d0c858-c6bd-4ff3-a60e-10932174fd12-iptables-alerter-script\") pod \"iptables-alerter-92748\" (UID: \"a6d0c858-c6bd-4ff3-a60e-10932174fd12\") " pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-slash\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/32d800a9-78b7-4d36-9616-e8f99b42df1f-ovnkube-config\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-registration-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c5faf65-3a14-47b8-8077-af57e33f8dfc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-socket-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-lib-modules\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-cni-dir\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-daemon-config\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-kubelet-config\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj5jc\" (UniqueName: \"kubernetes.io/projected/ddb0ced7-abc3-439b-b250-ff90f01679cc-kube-api-access-dj5jc\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c5faf65-3a14-47b8-8077-af57e33f8dfc-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-cni-netd\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.830701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33989fb0-89ab-47a8-90a4-2fbd2332d884-host\") pod \"node-ca-22cxn\" (UID: \"33989fb0-89ab-47a8-90a4-2fbd2332d884\") " pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c5faf65-3a14-47b8-8077-af57e33f8dfc-cnibin\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-run\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a2a1999-931a-46a1-83b0-57b1c9ca0dda-konnectivity-ca\") pod \"konnectivity-agent-r9hm4\" (UID: \"1a2a1999-931a-46a1-83b0-57b1c9ca0dda\") " pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-host\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-run-netns\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-host-run-netns\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73d5e141-cf74-4545-91ef-0e34b3c8b849-registration-dir\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.828958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-slash\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.829009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-cni-dir\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.829032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/32d800a9-78b7-4d36-9616-e8f99b42df1f-ovnkube-config\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.829054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/32d800a9-78b7-4d36-9616-e8f99b42df1f-host-cni-netd\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.829109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-kubelet-config\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:03.831622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.829391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-multus-daemon-config\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.832336 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.831748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/32d800a9-78b7-4d36-9616-e8f99b42df1f-ovn-node-metrics-cert\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.833474 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.833447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a2a1999-931a-46a1-83b0-57b1c9ca0dda-agent-certs\") pod \"konnectivity-agent-r9hm4\" (UID: \"1a2a1999-931a-46a1-83b0-57b1c9ca0dda\") " pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:03.838634 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.838612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4f4v\" (UniqueName: \"kubernetes.io/projected/73d5e141-cf74-4545-91ef-0e34b3c8b849-kube-api-access-m4f4v\") pod \"aws-ebs-csi-driver-node-hskjw\" (UID: \"73d5e141-cf74-4545-91ef-0e34b3c8b849\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:03.839622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.839584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt9q6\" (UniqueName: \"kubernetes.io/projected/32d800a9-78b7-4d36-9616-e8f99b42df1f-kube-api-access-mt9q6\") pod \"ovnkube-node-t8kjk\" (UID: \"32d800a9-78b7-4d36-9616-e8f99b42df1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:03.841086 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.841061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvpj\" (UniqueName: \"kubernetes.io/projected/e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d-kube-api-access-pzvpj\") pod \"multus-zdnlz\" (UID: \"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d\") " pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:03.890601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.890574    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 01:22:03.919628 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.919571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal" event={"ID":"cff2e4f35896379a5eb0549eb6d3cc94","Type":"ContainerStarted","Data":"eef4d18f56589de7cdac7e21810fb65d3621255801e1a44c9fdcd9888badce64"}
Sep 17 01:22:03.921220 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.921171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal" event={"ID":"e047cec48a9cc0053170912f91d4d94a","Type":"ContainerStarted","Data":"1bcacf2335fa6e319176c11917a1a0595b8e3d01266b7f2526b42367ef4f60e5"}
Sep 17 01:22:03.929223 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ptd\" (UniqueName: \"kubernetes.io/projected/a6d0c858-c6bd-4ff3-a60e-10932174fd12-kube-api-access-l7ptd\") pod \"iptables-alerter-92748\" (UID: \"a6d0c858-c6bd-4ff3-a60e-10932174fd12\") " pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:03.929316 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpgm\" (UniqueName: \"kubernetes.io/projected/d2d5b623-d9d2-4621-bc65-66241a002022-kube-api-access-dkpgm\") pod \"node-resolver-mxzkv\" (UID: \"d2d5b623-d9d2-4621-bc65-66241a002022\") " pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:03.929447 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-sysctl-conf\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.929496 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d5b623-d9d2-4621-bc65-66241a002022-tmp-dir\") pod \"node-resolver-mxzkv\" (UID: \"d2d5b623-d9d2-4621-bc65-66241a002022\") " pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:03.929534 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-modprobe-d\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.929534 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-systemd\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.929652 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2d5b623-d9d2-4621-bc65-66241a002022-hosts-file\") pod \"node-resolver-mxzkv\" (UID: \"d2d5b623-d9d2-4621-bc65-66241a002022\") " pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:03.929652 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c5faf65-3a14-47b8-8077-af57e33f8dfc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.929652 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cfhj\" (UniqueName: \"kubernetes.io/projected/7c5faf65-3a14-47b8-8077-af57e33f8dfc-kube-api-access-9cfhj\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.929762 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-tuned\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.929762 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c5faf65-3a14-47b8-8077-af57e33f8dfc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.929762 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-sysctl-conf\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.929762 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07550c1-c7c1-4168-82fe-0e00fe3221dd-tmp\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.929762 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a6d0c858-c6bd-4ff3-a60e-10932174fd12-iptables-alerter-script\") pod \"iptables-alerter-92748\" (UID: \"a6d0c858-c6bd-4ff3-a60e-10932174fd12\") " pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:03.929762 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2d5b623-d9d2-4621-bc65-66241a002022-hosts-file\") pod \"node-resolver-mxzkv\" (UID: \"d2d5b623-d9d2-4621-bc65-66241a002022\") " pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:03.930034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c5faf65-3a14-47b8-8077-af57e33f8dfc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.930034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-lib-modules\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.930034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj5jc\" (UniqueName: \"kubernetes.io/projected/ddb0ced7-abc3-439b-b250-ff90f01679cc-kube-api-access-dj5jc\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:03.930034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c5faf65-3a14-47b8-8077-af57e33f8dfc-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.930034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33989fb0-89ab-47a8-90a4-2fbd2332d884-host\") pod \"node-ca-22cxn\" (UID: \"33989fb0-89ab-47a8-90a4-2fbd2332d884\") " pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:03.930034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c5faf65-3a14-47b8-8077-af57e33f8dfc-cnibin\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.930034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-run\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.930034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-host\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.930034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.929980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-sysctl-d\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.930492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33989fb0-89ab-47a8-90a4-2fbd2332d884-host\") pod \"node-ca-22cxn\" (UID: \"33989fb0-89ab-47a8-90a4-2fbd2332d884\") " pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:03.930492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d5b623-d9d2-4621-bc65-66241a002022-tmp-dir\") pod \"node-resolver-mxzkv\" (UID: \"d2d5b623-d9d2-4621-bc65-66241a002022\") " pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:03.930492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-systemd\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.930492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-sysctl-d\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.930492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-modprobe-d\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.930492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-lib-modules\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.930492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-run\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.930492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-host\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.930492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c5faf65-3a14-47b8-8077-af57e33f8dfc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.930492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c5faf65-3a14-47b8-8077-af57e33f8dfc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c5faf65-3a14-47b8-8077-af57e33f8dfc-cnibin\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nw5q\" (UniqueName: \"kubernetes.io/projected/a07550c1-c7c1-4168-82fe-0e00fe3221dd-kube-api-access-7nw5q\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c5faf65-3a14-47b8-8077-af57e33f8dfc-system-cni-dir\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-var-lib-kubelet\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33989fb0-89ab-47a8-90a4-2fbd2332d884-serviceca\") pod \"node-ca-22cxn\" (UID: \"33989fb0-89ab-47a8-90a4-2fbd2332d884\") " pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-sysconfig\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-kubernetes\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-sys\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-var-lib-kubelet\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486xp\" (UniqueName: \"kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp\") pod \"network-check-target-jsnp2\" (UID: \"1026d5e4-b4fd-4730-b074-7973a88b774c\") " pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c5faf65-3a14-47b8-8077-af57e33f8dfc-system-cni-dir\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-kubernetes\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6d0c858-c6bd-4ff3-a60e-10932174fd12-host-slash\") pod \"iptables-alerter-92748\" (UID: \"a6d0c858-c6bd-4ff3-a60e-10932174fd12\") " pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a6d0c858-c6bd-4ff3-a60e-10932174fd12-iptables-alerter-script\") pod \"iptables-alerter-92748\" (UID: \"a6d0c858-c6bd-4ff3-a60e-10932174fd12\") " pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c5faf65-3a14-47b8-8077-af57e33f8dfc-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.931039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c5faf65-3a14-47b8-8077-af57e33f8dfc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:03.931906 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.930930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frh2m\" (UniqueName: \"kubernetes.io/projected/33989fb0-89ab-47a8-90a4-2fbd2332d884-kube-api-access-frh2m\") pod \"node-ca-22cxn\" (UID: \"33989fb0-89ab-47a8-90a4-2fbd2332d884\") " pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:03.931906 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.931023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6d0c858-c6bd-4ff3-a60e-10932174fd12-host-slash\") pod \"iptables-alerter-92748\" (UID: \"a6d0c858-c6bd-4ff3-a60e-10932174fd12\") " pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:03.931906 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.931036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-sysconfig\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.931906 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.931098    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:03.931906 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.931165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a07550c1-c7c1-4168-82fe-0e00fe3221dd-sys\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.931906 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.931170    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs podName:ddb0ced7-abc3-439b-b250-ff90f01679cc nodeName:}" failed. No retries permitted until 2026-09-17 01:22:04.431137995 +0000 UTC m=+3.101654085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs") pod "network-metrics-daemon-blh4d" (UID: "ddb0ced7-abc3-439b-b250-ff90f01679cc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:03.931906 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.931327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33989fb0-89ab-47a8-90a4-2fbd2332d884-serviceca\") pod \"node-ca-22cxn\" (UID: \"33989fb0-89ab-47a8-90a4-2fbd2332d884\") " pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:03.933873 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.933790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07550c1-c7c1-4168-82fe-0e00fe3221dd-tmp\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.933873 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.933855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a07550c1-c7c1-4168-82fe-0e00fe3221dd-etc-tuned\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.956655 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.956575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ptd\" (UniqueName: \"kubernetes.io/projected/a6d0c858-c6bd-4ff3-a60e-10932174fd12-kube-api-access-l7ptd\") pod \"iptables-alerter-92748\" (UID: \"a6d0c858-c6bd-4ff3-a60e-10932174fd12\") " pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:03.957228 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.957179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpgm\" (UniqueName: \"kubernetes.io/projected/d2d5b623-d9d2-4621-bc65-66241a002022-kube-api-access-dkpgm\") pod \"node-resolver-mxzkv\" (UID: \"d2d5b623-d9d2-4621-bc65-66241a002022\") " pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:03.965122 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.965102    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 01:22:03.965122 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.965125    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 01:22:03.965297 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.965141    2578 projected.go:196] Error preparing data for projected volume kube-api-access-486xp for pod openshift-network-diagnostics/network-check-target-jsnp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:03.965297 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:03.965218    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp podName:1026d5e4-b4fd-4730-b074-7973a88b774c nodeName:}" failed. No retries permitted until 2026-09-17 01:22:04.465200321 +0000 UTC m=+3.135716392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-486xp" (UniqueName: "kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp") pod "network-check-target-jsnp2" (UID: "1026d5e4-b4fd-4730-b074-7973a88b774c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:03.966343 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.966311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj5jc\" (UniqueName: \"kubernetes.io/projected/ddb0ced7-abc3-439b-b250-ff90f01679cc-kube-api-access-dj5jc\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:03.966759 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.966739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frh2m\" (UniqueName: \"kubernetes.io/projected/33989fb0-89ab-47a8-90a4-2fbd2332d884-kube-api-access-frh2m\") pod \"node-ca-22cxn\" (UID: \"33989fb0-89ab-47a8-90a4-2fbd2332d884\") " pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:03.967732 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.967694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nw5q\" (UniqueName: \"kubernetes.io/projected/a07550c1-c7c1-4168-82fe-0e00fe3221dd-kube-api-access-7nw5q\") pod \"tuned-c6xsj\" (UID: \"a07550c1-c7c1-4168-82fe-0e00fe3221dd\") " pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:03.968233 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:03.968208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cfhj\" (UniqueName: \"kubernetes.io/projected/7c5faf65-3a14-47b8-8077-af57e33f8dfc-kube-api-access-9cfhj\") pod \"multus-additional-cni-plugins-hvtgt\" (UID: \"7c5faf65-3a14-47b8-8077-af57e33f8dfc\") " pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:04.016444 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.016408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw"
Sep 17 01:22:04.025550 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.025522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zdnlz"
Sep 17 01:22:04.036761 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.036740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:04.045752 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.045733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:04.054616 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.054597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-22cxn"
Sep 17 01:22:04.064591 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.064573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-92748"
Sep 17 01:22:04.073489 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.073468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mxzkv"
Sep 17 01:22:04.081396 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.081377    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hvtgt"
Sep 17 01:22:04.088346 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.088327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c6xsj"
Sep 17 01:22:04.165217 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.165192    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 01:22:04.333787 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.333701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:04.333954 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:04.333882    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:04.334014 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:04.333966    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret podName:a9850b1d-04e3-4bbd-9072-a9ac2dab518a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:05.333943506 +0000 UTC m=+4.004459574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret") pod "global-pull-secret-syncer-mws4d" (UID: "a9850b1d-04e3-4bbd-9072-a9ac2dab518a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:04.434835 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.434805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:04.434990 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:04.434930    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:04.435032 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:04.434990    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs podName:ddb0ced7-abc3-439b-b250-ff90f01679cc nodeName:}" failed. No retries permitted until 2026-09-17 01:22:05.434974589 +0000 UTC m=+4.105490680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs") pod "network-metrics-daemon-blh4d" (UID: "ddb0ced7-abc3-439b-b250-ff90f01679cc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:04.535505 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.535467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486xp\" (UniqueName: \"kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp\") pod \"network-check-target-jsnp2\" (UID: \"1026d5e4-b4fd-4730-b074-7973a88b774c\") " pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:04.535681 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:04.535631    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 01:22:04.535681 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:04.535645    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 01:22:04.535681 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:04.535655    2578 projected.go:196] Error preparing data for projected volume kube-api-access-486xp for pod openshift-network-diagnostics/network-check-target-jsnp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:04.535823 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:04.535710    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp podName:1026d5e4-b4fd-4730-b074-7973a88b774c nodeName:}" failed. No retries permitted until 2026-09-17 01:22:05.535695369 +0000 UTC m=+4.206211437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-486xp" (UniqueName: "kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp") pod "network-check-target-jsnp2" (UID: "1026d5e4-b4fd-4730-b074-7973a88b774c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:04.759981 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.759888    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 01:17:04 +0000 UTC" deadline="2028-05-27 16:27:39.762132722 +0000 UTC"
Sep 17 01:22:04.759981 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:04.759928    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14847h5m35.00220943s"
Sep 17 01:22:05.070798 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:05.070669    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07550c1_c7c1_4168_82fe_0e00fe3221dd.slice/crio-4ca375e9b3f7f91ea902788d65746d527ad76f2bf15926adbfb93814e7737eeb WatchSource:0}: Error finding container 4ca375e9b3f7f91ea902788d65746d527ad76f2bf15926adbfb93814e7737eeb: Status 404 returned error can't find the container with id 4ca375e9b3f7f91ea902788d65746d527ad76f2bf15926adbfb93814e7737eeb
Sep 17 01:22:05.077966 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:05.077942    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d0c858_c6bd_4ff3_a60e_10932174fd12.slice/crio-88a91f6225bda56d7161ebafac137c01a759293d75e22db657aba6f620c3c524 WatchSource:0}: Error finding container 88a91f6225bda56d7161ebafac137c01a759293d75e22db657aba6f620c3c524: Status 404 returned error can't find the container with id 88a91f6225bda56d7161ebafac137c01a759293d75e22db657aba6f620c3c524
Sep 17 01:22:05.079399 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:05.079004    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d5b623_d9d2_4621_bc65_66241a002022.slice/crio-2c3d615d3d477c1da7167169ffa86d671f815d4060862a56a0f1dd185e4d5b45 WatchSource:0}: Error finding container 2c3d615d3d477c1da7167169ffa86d671f815d4060862a56a0f1dd185e4d5b45: Status 404 returned error can't find the container with id 2c3d615d3d477c1da7167169ffa86d671f815d4060862a56a0f1dd185e4d5b45
Sep 17 01:22:05.080014 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:05.079952    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5faf65_3a14_47b8_8077_af57e33f8dfc.slice/crio-35add912411b9d1f991bae7eaa24e13921ca00a35520aa5d503be270e29ce9af WatchSource:0}: Error finding container 35add912411b9d1f991bae7eaa24e13921ca00a35520aa5d503be270e29ce9af: Status 404 returned error can't find the container with id 35add912411b9d1f991bae7eaa24e13921ca00a35520aa5d503be270e29ce9af
Sep 17 01:22:05.081005 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:05.080977    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d800a9_78b7_4d36_9616_e8f99b42df1f.slice/crio-9d4dca613f89edca72b51f809453ac47dbce7f56ff596f5705e17ef3357f2c4c WatchSource:0}: Error finding container 9d4dca613f89edca72b51f809453ac47dbce7f56ff596f5705e17ef3357f2c4c: Status 404 returned error can't find the container with id 9d4dca613f89edca72b51f809453ac47dbce7f56ff596f5705e17ef3357f2c4c
Sep 17 01:22:05.081220 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:05.081179    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33989fb0_89ab_47a8_90a4_2fbd2332d884.slice/crio-6934144d13780604fd30b3adc9348d37332cbf0f538d92e309091b3577ebe3c2 WatchSource:0}: Error finding container 6934144d13780604fd30b3adc9348d37332cbf0f538d92e309091b3577ebe3c2: Status 404 returned error can't find the container with id 6934144d13780604fd30b3adc9348d37332cbf0f538d92e309091b3577ebe3c2
Sep 17 01:22:05.083183 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:05.083107    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d5e141_cf74_4545_91ef_0e34b3c8b849.slice/crio-e2699fd2e3f6fd3acf0f4e873432c3dfd62ada71c02fe5bfea0709a86fedcc08 WatchSource:0}: Error finding container e2699fd2e3f6fd3acf0f4e873432c3dfd62ada71c02fe5bfea0709a86fedcc08: Status 404 returned error can't find the container with id e2699fd2e3f6fd3acf0f4e873432c3dfd62ada71c02fe5bfea0709a86fedcc08
Sep 17 01:22:05.084052 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:05.084023    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ba1fbb_6994_4cd1_a0db_2ac0299ef57d.slice/crio-d763724db0db2ca1da084ed861b1e844a783a728fe2bc15830cb626ffef895e3 WatchSource:0}: Error finding container d763724db0db2ca1da084ed861b1e844a783a728fe2bc15830cb626ffef895e3: Status 404 returned error can't find the container with id d763724db0db2ca1da084ed861b1e844a783a728fe2bc15830cb626ffef895e3
Sep 17 01:22:05.087066 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:05.086678    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2a1999_931a_46a1_83b0_57b1c9ca0dda.slice/crio-d684848a2d452bb40b9dfc236d2cd4a221c567ad6cabde14efd6a51792477891 WatchSource:0}: Error finding container d684848a2d452bb40b9dfc236d2cd4a221c567ad6cabde14efd6a51792477891: Status 404 returned error can't find the container with id d684848a2d452bb40b9dfc236d2cd4a221c567ad6cabde14efd6a51792477891
Sep 17 01:22:05.342257 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.342067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:05.342386 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.342211    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:05.342426 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.342392    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret podName:a9850b1d-04e3-4bbd-9072-a9ac2dab518a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:07.342375366 +0000 UTC m=+6.012891449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret") pod "global-pull-secret-syncer-mws4d" (UID: "a9850b1d-04e3-4bbd-9072-a9ac2dab518a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:05.443060 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.443028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:05.443223 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.443198    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:05.443283 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.443272    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs podName:ddb0ced7-abc3-439b-b250-ff90f01679cc nodeName:}" failed. No retries permitted until 2026-09-17 01:22:07.443256343 +0000 UTC m=+6.113772412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs") pod "network-metrics-daemon-blh4d" (UID: "ddb0ced7-abc3-439b-b250-ff90f01679cc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:05.543885 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.543854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486xp\" (UniqueName: \"kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp\") pod \"network-check-target-jsnp2\" (UID: \"1026d5e4-b4fd-4730-b074-7973a88b774c\") " pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:05.544071 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.544049    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 01:22:05.544141 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.544082    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 01:22:05.544141 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.544098    2578 projected.go:196] Error preparing data for projected volume kube-api-access-486xp for pod openshift-network-diagnostics/network-check-target-jsnp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:05.544256 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.544170    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp podName:1026d5e4-b4fd-4730-b074-7973a88b774c nodeName:}" failed. No retries permitted until 2026-09-17 01:22:07.544145193 +0000 UTC m=+6.214661286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-486xp" (UniqueName: "kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp") pod "network-check-target-jsnp2" (UID: "1026d5e4-b4fd-4730-b074-7973a88b774c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:05.760369 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.760249    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 01:17:04 +0000 UTC" deadline="2028-06-03 09:49:06.717794726 +0000 UTC"
Sep 17 01:22:05.760369 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.760314    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15008h27m0.957484642s"
Sep 17 01:22:05.915747 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.915713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:05.915937 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.915855    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:05.916288 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.916266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:05.916407 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.916384    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:05.918438 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.918414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:05.918555 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:05.918525    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:05.941304 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.941218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw" event={"ID":"73d5e141-cf74-4545-91ef-0e34b3c8b849","Type":"ContainerStarted","Data":"e2699fd2e3f6fd3acf0f4e873432c3dfd62ada71c02fe5bfea0709a86fedcc08"}
Sep 17 01:22:05.956441 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.956412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-92748" event={"ID":"a6d0c858-c6bd-4ff3-a60e-10932174fd12","Type":"ContainerStarted","Data":"88a91f6225bda56d7161ebafac137c01a759293d75e22db657aba6f620c3c524"}
Sep 17 01:22:05.962458 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.962430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zdnlz" event={"ID":"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d","Type":"ContainerStarted","Data":"d763724db0db2ca1da084ed861b1e844a783a728fe2bc15830cb626ffef895e3"}
Sep 17 01:22:05.972537 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.972505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-22cxn" event={"ID":"33989fb0-89ab-47a8-90a4-2fbd2332d884","Type":"ContainerStarted","Data":"6934144d13780604fd30b3adc9348d37332cbf0f538d92e309091b3577ebe3c2"}
Sep 17 01:22:05.976518 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.976491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk" event={"ID":"32d800a9-78b7-4d36-9616-e8f99b42df1f","Type":"ContainerStarted","Data":"9d4dca613f89edca72b51f809453ac47dbce7f56ff596f5705e17ef3357f2c4c"}
Sep 17 01:22:05.981246 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.981182    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvtgt" event={"ID":"7c5faf65-3a14-47b8-8077-af57e33f8dfc","Type":"ContainerStarted","Data":"35add912411b9d1f991bae7eaa24e13921ca00a35520aa5d503be270e29ce9af"}
Sep 17 01:22:05.985436 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.985360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mxzkv" event={"ID":"d2d5b623-d9d2-4621-bc65-66241a002022","Type":"ContainerStarted","Data":"2c3d615d3d477c1da7167169ffa86d671f815d4060862a56a0f1dd185e4d5b45"}
Sep 17 01:22:05.991021 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.990995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c6xsj" event={"ID":"a07550c1-c7c1-4168-82fe-0e00fe3221dd","Type":"ContainerStarted","Data":"4ca375e9b3f7f91ea902788d65746d527ad76f2bf15926adbfb93814e7737eeb"}
Sep 17 01:22:05.995749 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.995722    2578 generic.go:356] "Generic (PLEG): container finished" podID="cff2e4f35896379a5eb0549eb6d3cc94" containerID="afe92d8be74173e0fea47b21cfbb756532e63803cd986f67d6d834dd36020e03" exitCode=0
Sep 17 01:22:05.996141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:05.995889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal" event={"ID":"cff2e4f35896379a5eb0549eb6d3cc94","Type":"ContainerDied","Data":"afe92d8be74173e0fea47b21cfbb756532e63803cd986f67d6d834dd36020e03"}
Sep 17 01:22:06.007529 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:06.007457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal" event={"ID":"e047cec48a9cc0053170912f91d4d94a","Type":"ContainerStarted","Data":"6b947e209912160a506e0ac13ffb45a63555698b9023c290d6fe4a4027734d22"}
Sep 17 01:22:06.021100 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:06.021041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r9hm4" event={"ID":"1a2a1999-931a-46a1-83b0-57b1c9ca0dda","Type":"ContainerStarted","Data":"d684848a2d452bb40b9dfc236d2cd4a221c567ad6cabde14efd6a51792477891"}
Sep 17 01:22:07.045256 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:07.044785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal" event={"ID":"cff2e4f35896379a5eb0549eb6d3cc94","Type":"ContainerStarted","Data":"b9c77383c08d53e4e6322296a1e76fcba790163b0577ab91300cf24d642a4dc8"}
Sep 17 01:22:07.061660 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:07.061334    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-87.ec2.internal" podStartSLOduration=4.061311562 podStartE2EDuration="4.061311562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:06.024911465 +0000 UTC m=+4.695427558" watchObservedRunningTime="2026-09-17 01:22:07.061311562 +0000 UTC m=+5.731827656"
Sep 17 01:22:07.360754 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:07.360662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:07.360924 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.360823    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:07.360924 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.360895    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret podName:a9850b1d-04e3-4bbd-9072-a9ac2dab518a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:11.360874442 +0000 UTC m=+10.031390529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret") pod "global-pull-secret-syncer-mws4d" (UID: "a9850b1d-04e3-4bbd-9072-a9ac2dab518a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:07.461887 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:07.461271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:07.461887 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.461430    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:07.461887 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.461496    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs podName:ddb0ced7-abc3-439b-b250-ff90f01679cc nodeName:}" failed. No retries permitted until 2026-09-17 01:22:11.461476771 +0000 UTC m=+10.131992848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs") pod "network-metrics-daemon-blh4d" (UID: "ddb0ced7-abc3-439b-b250-ff90f01679cc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:07.562934 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:07.562225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486xp\" (UniqueName: \"kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp\") pod \"network-check-target-jsnp2\" (UID: \"1026d5e4-b4fd-4730-b074-7973a88b774c\") " pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:07.562934 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.562456    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 01:22:07.562934 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.562476    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 01:22:07.562934 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.562490    2578 projected.go:196] Error preparing data for projected volume kube-api-access-486xp for pod openshift-network-diagnostics/network-check-target-jsnp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:07.562934 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.562554    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp podName:1026d5e4-b4fd-4730-b074-7973a88b774c nodeName:}" failed. No retries permitted until 2026-09-17 01:22:11.562534872 +0000 UTC m=+10.233050945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-486xp" (UniqueName: "kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp") pod "network-check-target-jsnp2" (UID: "1026d5e4-b4fd-4730-b074-7973a88b774c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:07.914781 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:07.914746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:07.914988 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.914899    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:07.915382 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:07.915348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:07.915502 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.915472    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:07.915585 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:07.915531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:07.915644 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:07.915602    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:09.915152 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:09.915113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:09.915152 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:09.915115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:09.915746 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:09.915315    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:09.915746 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:09.915367    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:09.915746 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:09.915457    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:09.915930 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:09.915900    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:11.402761 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:11.402725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:11.403250 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.402897    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:11.403250 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.402980    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret podName:a9850b1d-04e3-4bbd-9072-a9ac2dab518a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:19.402956749 +0000 UTC m=+18.073472822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret") pod "global-pull-secret-syncer-mws4d" (UID: "a9850b1d-04e3-4bbd-9072-a9ac2dab518a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:11.503780 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:11.503742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:11.504028 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.503904    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:11.504028 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.503993    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs podName:ddb0ced7-abc3-439b-b250-ff90f01679cc nodeName:}" failed. No retries permitted until 2026-09-17 01:22:19.503972448 +0000 UTC m=+18.174488523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs") pod "network-metrics-daemon-blh4d" (UID: "ddb0ced7-abc3-439b-b250-ff90f01679cc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:11.604504 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:11.604466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486xp\" (UniqueName: \"kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp\") pod \"network-check-target-jsnp2\" (UID: \"1026d5e4-b4fd-4730-b074-7973a88b774c\") " pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:11.604829 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.604722    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 01:22:11.604829 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.604756    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 01:22:11.604829 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.604771    2578 projected.go:196] Error preparing data for projected volume kube-api-access-486xp for pod openshift-network-diagnostics/network-check-target-jsnp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:11.605014 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.604843    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp podName:1026d5e4-b4fd-4730-b074-7973a88b774c nodeName:}" failed. No retries permitted until 2026-09-17 01:22:19.604820263 +0000 UTC m=+18.275336349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-486xp" (UniqueName: "kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp") pod "network-check-target-jsnp2" (UID: "1026d5e4-b4fd-4730-b074-7973a88b774c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:11.915969 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:11.915930    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:11.916176 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.916066    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:11.916485 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:11.916464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:11.916609 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.916566    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:11.916719 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:11.916697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:11.916848 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:11.916787    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:13.914895 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:13.914858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:13.915274 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:13.914866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:13.915274 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:13.914993    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:13.915274 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:13.915006    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:13.915274 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:13.915124    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:13.915274 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:13.915201    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:15.914743 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:15.914707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:15.915189 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:15.914707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:15.915189 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:15.914875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:15.915189 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:15.914866    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:15.915189 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:15.914985    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:15.915189 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:15.915067    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:17.914315 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:17.914278    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:17.914810 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:17.914284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:17.914810 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:17.914405    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:17.914810 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:17.914420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:17.914810 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:17.914511    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:17.914810 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:17.914614    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:19.470982 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:19.470951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:19.471457 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.471118    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:19.471457 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.471194    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret podName:a9850b1d-04e3-4bbd-9072-a9ac2dab518a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:35.471172739 +0000 UTC m=+34.141688822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret") pod "global-pull-secret-syncer-mws4d" (UID: "a9850b1d-04e3-4bbd-9072-a9ac2dab518a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:19.571675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:19.571649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:19.571777 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.571762    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:19.571848 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.571823    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs podName:ddb0ced7-abc3-439b-b250-ff90f01679cc nodeName:}" failed. No retries permitted until 2026-09-17 01:22:35.571805082 +0000 UTC m=+34.242321151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs") pod "network-metrics-daemon-blh4d" (UID: "ddb0ced7-abc3-439b-b250-ff90f01679cc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:19.672794 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:19.672768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486xp\" (UniqueName: \"kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp\") pod \"network-check-target-jsnp2\" (UID: \"1026d5e4-b4fd-4730-b074-7973a88b774c\") " pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:19.672909 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.672900    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 01:22:19.672951 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.672912    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 01:22:19.672951 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.672922    2578 projected.go:196] Error preparing data for projected volume kube-api-access-486xp for pod openshift-network-diagnostics/network-check-target-jsnp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:19.673014 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.672968    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp podName:1026d5e4-b4fd-4730-b074-7973a88b774c nodeName:}" failed. No retries permitted until 2026-09-17 01:22:35.672955993 +0000 UTC m=+34.343472062 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-486xp" (UniqueName: "kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp") pod "network-check-target-jsnp2" (UID: "1026d5e4-b4fd-4730-b074-7973a88b774c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:19.914258 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:19.914176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:19.914417 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:19.914180    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:19.914417 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.914298    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:19.914417 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:19.914331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:19.914575 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.914435    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:19.914575 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:19.914523    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:21.919024 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:21.918963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:21.919550 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:21.919043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:21.919550 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:21.919077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:21.919550 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:21.919078    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:21.919550 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:21.919150    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:21.919550 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:21.919215    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:23.090587 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:23.088943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c6xsj" event={"ID":"a07550c1-c7c1-4168-82fe-0e00fe3221dd","Type":"ContainerStarted","Data":"b4c9a86b719f7b47ad9b702c3522a46dd0072ee02c5ffaa37561c0bd5c791764"}
Sep 17 01:22:23.104084 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:23.104031    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-87.ec2.internal" podStartSLOduration=20.104012491 podStartE2EDuration="20.104012491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:07.061557061 +0000 UTC m=+5.732073156" watchObservedRunningTime="2026-09-17 01:22:23.104012491 +0000 UTC m=+21.774528583"
Sep 17 01:22:23.104268 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:23.104238    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c6xsj" podStartSLOduration=3.218082257 podStartE2EDuration="21.104228942s" totalImagesPullingTime="17.886146685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:23.103634586 +0000 UTC m=+21.774150681" watchObservedRunningTime="2026-09-17 01:22:23.104228942 +0000 UTC m=+21.774745032"
Sep 17 01:22:23.915213 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:23.915000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:23.915382 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:23.915008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:23.915382 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:23.915235    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:23.915382 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:23.915311    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:23.915382 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:23.915083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:23.915676 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:23.915412    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:24.092424 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.092377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-22cxn" event={"ID":"33989fb0-89ab-47a8-90a4-2fbd2332d884","Type":"ContainerStarted","Data":"81839ed1cd4e19f6fa034cb5d97a758ff903da048a67b1938efe60d094728fec"}
Sep 17 01:22:24.095210 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.095176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk" event={"ID":"32d800a9-78b7-4d36-9616-e8f99b42df1f","Type":"ContainerStarted","Data":"1213b6f518a6df09211d23b46605abba47bfb3a72bd596a5aac854ef551cb60d"}
Sep 17 01:22:24.095330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.095256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk" event={"ID":"32d800a9-78b7-4d36-9616-e8f99b42df1f","Type":"ContainerStarted","Data":"63f36190716bfe3341ccac496776d2d32a8984eb620e548bdb828a6f9b19d513"}
Sep 17 01:22:24.095330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.095304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk" event={"ID":"32d800a9-78b7-4d36-9616-e8f99b42df1f","Type":"ContainerStarted","Data":"907c8ac3adaca0aacd2cb7c9d618662ac294960e1c0d8d01dde50d3b2189951f"}
Sep 17 01:22:24.095413 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.095349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk" event={"ID":"32d800a9-78b7-4d36-9616-e8f99b42df1f","Type":"ContainerStarted","Data":"d91f9c36ac184ac2b67085e1e7c68e44555a09208784a6f05dd394bfea173056"}
Sep 17 01:22:24.095413 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.095396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk" event={"ID":"32d800a9-78b7-4d36-9616-e8f99b42df1f","Type":"ContainerStarted","Data":"4f0cef82078ebe60d595a932182068c9709d1c1d4672e9bd13f3283fef2eb321"}
Sep 17 01:22:24.095495 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.095439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk" event={"ID":"32d800a9-78b7-4d36-9616-e8f99b42df1f","Type":"ContainerStarted","Data":"aab21c4be59545a2a30ef7e89edeab674e37778b67621cabd72cd7d7543f1ec7"}
Sep 17 01:22:24.096754 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.096725    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c5faf65-3a14-47b8-8077-af57e33f8dfc" containerID="e26ddb9f74f8be209afb5c776eeeed2974f6ec99c6c12f461b13863f6551958c" exitCode=0
Sep 17 01:22:24.096874 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.096756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvtgt" event={"ID":"7c5faf65-3a14-47b8-8077-af57e33f8dfc","Type":"ContainerDied","Data":"e26ddb9f74f8be209afb5c776eeeed2974f6ec99c6c12f461b13863f6551958c"}
Sep 17 01:22:24.098294 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.098275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mxzkv" event={"ID":"d2d5b623-d9d2-4621-bc65-66241a002022","Type":"ContainerStarted","Data":"91bd52601035dbfa8283541bc3dd7a33bd2cca56cf5e9f4e6a3299870df8e9b3"}
Sep 17 01:22:24.099751 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.099731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r9hm4" event={"ID":"1a2a1999-931a-46a1-83b0-57b1c9ca0dda","Type":"ContainerStarted","Data":"a424833e422094fd8cc321b34098f9d213a1e60bee60533d6edb10adc0e41e8b"}
Sep 17 01:22:24.101083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.101065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw" event={"ID":"73d5e141-cf74-4545-91ef-0e34b3c8b849","Type":"ContainerStarted","Data":"80a1e8d7a7389cda89e1b57ae15a2407509a5027eb9d797d9231117b27857195"}
Sep 17 01:22:24.102474 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.102452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zdnlz" event={"ID":"e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d","Type":"ContainerStarted","Data":"da995da92d69718b50079fefce6564e4ebd32093f6cc3c5c86bd1f2a2d29aec5"}
Sep 17 01:22:24.105364 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.105331    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-22cxn" podStartSLOduration=4.231911945 podStartE2EDuration="22.105319659s" totalImagesPullingTime="17.873407714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:24.105035809 +0000 UTC m=+22.775551897" watchObservedRunningTime="2026-09-17 01:22:24.105319659 +0000 UTC m=+22.775835739"
Sep 17 01:22:24.117579 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.117526    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mxzkv" podStartSLOduration=4.243424865 podStartE2EDuration="22.117514052s" totalImagesPullingTime="17.874089187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:24.117379537 +0000 UTC m=+22.787895627" watchObservedRunningTime="2026-09-17 01:22:24.117514052 +0000 UTC m=+22.788030141"
Sep 17 01:22:24.131729 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.131698    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zdnlz" podStartSLOduration=4.211394281 podStartE2EDuration="22.131687624s" totalImagesPullingTime="17.920293343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:24.131449964 +0000 UTC m=+22.801966053" watchObservedRunningTime="2026-09-17 01:22:24.131687624 +0000 UTC m=+22.802203713"
Sep 17 01:22:24.143966 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.143923    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r9hm4" podStartSLOduration=7.752499589 podStartE2EDuration="22.143912544s" totalImagesPullingTime="14.391412955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:24.143307778 +0000 UTC m=+22.813823889" watchObservedRunningTime="2026-09-17 01:22:24.143912544 +0000 UTC m=+22.814428632"
Sep 17 01:22:24.269447 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.269418    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 01:22:24.782961 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.782834    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T01:22:24.269437228Z","UUID":"b6a6ed25-1bf9-401b-bb07-1de2ac48697d","Handler":null,"Name":"","Endpoint":""}
Sep 17 01:22:24.784462 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.784434    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 01:22:24.784602 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:24.784472    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 01:22:25.107217 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:25.107137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw" event={"ID":"73d5e141-cf74-4545-91ef-0e34b3c8b849","Type":"ContainerStarted","Data":"a7f79816058a1e53e250ff15ed1c93d11fce030322befb7ffdf66cbb894d24c5"}
Sep 17 01:22:25.108882 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:25.108858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-92748" event={"ID":"a6d0c858-c6bd-4ff3-a60e-10932174fd12","Type":"ContainerStarted","Data":"ccb1f0a117819d3d790c480d7ad6387cafc0f3a37ecca9ab2352febbb30f4762"}
Sep 17 01:22:25.122871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:25.122823    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-92748" podStartSLOduration=8.722843931 podStartE2EDuration="23.122806418s" totalImagesPullingTime="14.399962487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:25.122709003 +0000 UTC m=+23.793225093" watchObservedRunningTime="2026-09-17 01:22:25.122806418 +0000 UTC m=+23.793322508"
Sep 17 01:22:25.198648 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:25.198616    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:25.915081 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:25.914996    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:25.915081 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:25.915038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:25.915292 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:25.914996    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:25.915292 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:25.915117    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:25.915292 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:25.915227    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:25.915431 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:25.915326    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:26.114155 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:26.114117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw" event={"ID":"73d5e141-cf74-4545-91ef-0e34b3c8b849","Type":"ContainerStarted","Data":"044753353d58c21ee186cc8edb8950ffe87e63e665715732a18cfdeafa3b0f6e"}
Sep 17 01:22:26.118204 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:26.118180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk" event={"ID":"32d800a9-78b7-4d36-9616-e8f99b42df1f","Type":"ContainerStarted","Data":"5a2204b1e8b47f8f8b1858d0a56b884d79789eb84d98e926638292171686c533"}
Sep 17 01:22:26.130126 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:26.130084    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hskjw" podStartSLOduration=4.355330227 podStartE2EDuration="24.13007089s" totalImagesPullingTime="19.774740663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:26.129667789 +0000 UTC m=+24.800183888" watchObservedRunningTime="2026-09-17 01:22:26.13007089 +0000 UTC m=+24.800587020"
Sep 17 01:22:27.914726 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:27.914553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:27.915158 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:27.914561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:27.915158 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:27.914815    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:27.915158 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:27.914561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:27.915158 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:27.914891    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:27.915158 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:27.914947    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:28.126837 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:28.126762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk" event={"ID":"32d800a9-78b7-4d36-9616-e8f99b42df1f","Type":"ContainerStarted","Data":"bca13db916c9c765bc5597e6fe73b178797d331a83441017cb0ab989acbf5c49"}
Sep 17 01:22:28.127133 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:28.126976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:28.146303 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:28.146278    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:28.150838 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:28.150799    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk" podStartSLOduration=8.212251444 podStartE2EDuration="26.150787023s" totalImagesPullingTime="17.938535579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:28.14977206 +0000 UTC m=+26.820288148" watchObservedRunningTime="2026-09-17 01:22:28.150787023 +0000 UTC m=+26.821303109"
Sep 17 01:22:28.897174 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:28.897143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:28.897782 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:28.897762    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:29.129647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:29.129244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:29.129647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:29.129351    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:29.130142 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:29.129905    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r9hm4"
Sep 17 01:22:29.146636 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:29.146609    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:22:29.632379 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:29.632178    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mws4d"]
Sep 17 01:22:29.632597 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:29.632487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:29.632673 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:29.632598    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:29.635638 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:29.635481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jsnp2"]
Sep 17 01:22:29.635638 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:29.635605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:29.635812 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:29.635692    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:29.636232 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:29.636181    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-blh4d"]
Sep 17 01:22:29.636347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:29.636284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:29.636419 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:29.636381    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:30.914287 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:30.914219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:30.914934 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:30.914220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:30.914934 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:30.914329    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:30.914934 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:30.914388    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:31.134999 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:31.134968    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c5faf65-3a14-47b8-8077-af57e33f8dfc" containerID="1c0e938b1d72486d9a0c25971a00e0e38792b3841beb1e9f7479ec335c072f00" exitCode=0
Sep 17 01:22:31.135176 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:31.135041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvtgt" event={"ID":"7c5faf65-3a14-47b8-8077-af57e33f8dfc","Type":"ContainerDied","Data":"1c0e938b1d72486d9a0c25971a00e0e38792b3841beb1e9f7479ec335c072f00"}
Sep 17 01:22:31.915186 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:31.915149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:31.915822 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:31.915239    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:32.914458 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:32.914288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:32.914668 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:32.914297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:32.914668 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:32.914520    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:32.914668 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:32.914619    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:33.141347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:33.141319    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c5faf65-3a14-47b8-8077-af57e33f8dfc" containerID="545ca3403a80dc95f21a47a7e3309050996ba732f03aed8b623d5961d23e7717" exitCode=0
Sep 17 01:22:33.141707 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:33.141368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvtgt" event={"ID":"7c5faf65-3a14-47b8-8077-af57e33f8dfc","Type":"ContainerDied","Data":"545ca3403a80dc95f21a47a7e3309050996ba732f03aed8b623d5961d23e7717"}
Sep 17 01:22:33.914176 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:33.914149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:33.914412 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:33.914269    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:22:34.914860 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:34.914819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:34.915253 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:34.914871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:34.915253 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:34.914936    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsnp2" podUID="1026d5e4-b4fd-4730-b074-7973a88b774c"
Sep 17 01:22:34.915253 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:34.915040    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mws4d" podUID="a9850b1d-04e3-4bbd-9072-a9ac2dab518a"
Sep 17 01:22:35.147624 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.147592    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c5faf65-3a14-47b8-8077-af57e33f8dfc" containerID="b9d56030f7cdfbac28ec96846f6b847a9c06e185de3fdab49f762ecfe564550c" exitCode=0
Sep 17 01:22:35.147624 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.147628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvtgt" event={"ID":"7c5faf65-3a14-47b8-8077-af57e33f8dfc","Type":"ContainerDied","Data":"b9d56030f7cdfbac28ec96846f6b847a9c06e185de3fdab49f762ecfe564550c"}
Sep 17 01:22:35.492214 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.492185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:35.492344 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.492318    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:35.492384 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.492376    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret podName:a9850b1d-04e3-4bbd-9072-a9ac2dab518a nodeName:}" failed. No retries permitted until 2026-09-17 01:23:07.492362556 +0000 UTC m=+66.162878623 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret") pod "global-pull-secret-syncer-mws4d" (UID: "a9850b1d-04e3-4bbd-9072-a9ac2dab518a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 01:22:35.569806 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.569786    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-87.ec2.internal" event="NodeReady"
Sep 17 01:22:35.569928 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.569899    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 01:22:35.592968 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.592945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:35.593087 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.593064    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:35.593123 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.593109    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs podName:ddb0ced7-abc3-439b-b250-ff90f01679cc nodeName:}" failed. No retries permitted until 2026-09-17 01:23:07.593096792 +0000 UTC m=+66.263612861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs") pod "network-metrics-daemon-blh4d" (UID: "ddb0ced7-abc3-439b-b250-ff90f01679cc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 01:22:35.601576 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.601552    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"]
Sep 17 01:22:35.603366 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.603353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.606858 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.606475    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dn79c\""
Sep 17 01:22:35.606858 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.606643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 01:22:35.606858 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.606730    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 01:22:35.607126 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.607103    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 01:22:35.611856 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.611836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 01:22:35.627825 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.627775    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"]
Sep 17 01:22:35.628297 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.628279    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nrzkz"]
Sep 17 01:22:35.630220 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.630201    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pdvgq"]
Sep 17 01:22:35.630338 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.630324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.631924 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.631908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pdvgq"
Sep 17 01:22:35.632915 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.632898    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 01:22:35.633172 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.633157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sdvq5\""
Sep 17 01:22:35.633386 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.633373    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 01:22:35.634259 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.634239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 01:22:35.634514 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.634336    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 01:22:35.634514 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.634465    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 01:22:35.634728 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.634710    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mtkb2\""
Sep 17 01:22:35.650340 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.650314    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pdvgq"]
Sep 17 01:22:35.651781 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.651721    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nrzkz"]
Sep 17 01:22:35.693871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.693833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-trusted-ca\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.694037 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.693904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486xp\" (UniqueName: \"kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp\") pod \"network-check-target-jsnp2\" (UID: \"1026d5e4-b4fd-4730-b074-7973a88b774c\") " pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:35.694037 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.693933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-ca-trust-extracted\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.694037 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.693977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-certificates\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.694213 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.694082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-installation-pull-secrets\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.694213 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.694088    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 01:22:35.694213 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.694112    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 01:22:35.694213 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.694119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxzgv\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-kube-api-access-dxzgv\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.694213 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.694127    2578 projected.go:196] Error preparing data for projected volume kube-api-access-486xp for pod openshift-network-diagnostics/network-check-target-jsnp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:35.694213 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.694148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.694213 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.694192    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp podName:1026d5e4-b4fd-4730-b074-7973a88b774c nodeName:}" failed. No retries permitted until 2026-09-17 01:23:07.694174312 +0000 UTC m=+66.364690385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-486xp" (UniqueName: "kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp") pod "network-check-target-jsnp2" (UID: "1026d5e4-b4fd-4730-b074-7973a88b774c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 01:22:35.694621 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.694216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-image-registry-private-configuration\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.694621 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.694256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-bound-sa-token\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.794868 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.794837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-certificates\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.795029 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.794879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-installation-pull-secrets\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.795190 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxzgv\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-kube-api-access-dxzgv\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.795256 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.795256 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.795256 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123f13ef-59c4-4635-8550-ee114b9d1b7b-tmp-dir\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.795411 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff7ct\" (UniqueName: \"kubernetes.io/projected/123f13ef-59c4-4635-8550-ee114b9d1b7b-kube-api-access-ff7ct\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.795411 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-image-registry-private-configuration\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.795411 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/123f13ef-59c4-4635-8550-ee114b9d1b7b-config-volume\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.795411 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-bound-sa-token\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.795632 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdbc\" (UniqueName: \"kubernetes.io/projected/c7e25797-9a29-4a55-b0d9-bb5b89699bae-kube-api-access-tpdbc\") pod \"ingress-canary-pdvgq\" (UID: \"c7e25797-9a29-4a55-b0d9-bb5b89699bae\") " pod="openshift-ingress-canary/ingress-canary-pdvgq"
Sep 17 01:22:35.795632 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-trusted-ca\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.795632 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7e25797-9a29-4a55-b0d9-bb5b89699bae-cert\") pod \"ingress-canary-pdvgq\" (UID: \"c7e25797-9a29-4a55-b0d9-bb5b89699bae\") " pod="openshift-ingress-canary/ingress-canary-pdvgq"
Sep 17 01:22:35.795632 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-ca-trust-extracted\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.795632 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-certificates\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.795872 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.795811    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 01:22:35.795872 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.795831    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c: secret "image-registry-tls" not found
Sep 17 01:22:35.795971 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.795887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-ca-trust-extracted\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.795971 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.795908    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls podName:4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:36.295889003 +0000 UTC m=+34.966405085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls") pod "image-registry-7b4c7cb8b4-tfr4c" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a") : secret "image-registry-tls" not found
Sep 17 01:22:35.796447 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.796417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-trusted-ca\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.799613 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.799589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-installation-pull-secrets\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.799613 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.799608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-image-registry-private-configuration\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.804184 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.804114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxzgv\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-kube-api-access-dxzgv\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.804184 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.804121    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-bound-sa-token\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:35.896248 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.896203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/123f13ef-59c4-4635-8550-ee114b9d1b7b-config-volume\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.896248 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.896254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdbc\" (UniqueName: \"kubernetes.io/projected/c7e25797-9a29-4a55-b0d9-bb5b89699bae-kube-api-access-tpdbc\") pod \"ingress-canary-pdvgq\" (UID: \"c7e25797-9a29-4a55-b0d9-bb5b89699bae\") " pod="openshift-ingress-canary/ingress-canary-pdvgq"
Sep 17 01:22:35.896561 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.896366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7e25797-9a29-4a55-b0d9-bb5b89699bae-cert\") pod \"ingress-canary-pdvgq\" (UID: \"c7e25797-9a29-4a55-b0d9-bb5b89699bae\") " pod="openshift-ingress-canary/ingress-canary-pdvgq"
Sep 17 01:22:35.896561 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.896457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.896561 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.896490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123f13ef-59c4-4635-8550-ee114b9d1b7b-tmp-dir\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.896561 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.896507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff7ct\" (UniqueName: \"kubernetes.io/projected/123f13ef-59c4-4635-8550-ee114b9d1b7b-kube-api-access-ff7ct\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.896561 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.896562    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 01:22:35.896814 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:35.896610    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls podName:123f13ef-59c4-4635-8550-ee114b9d1b7b nodeName:}" failed. No retries permitted until 2026-09-17 01:22:36.396593252 +0000 UTC m=+35.067109319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls") pod "dns-default-nrzkz" (UID: "123f13ef-59c4-4635-8550-ee114b9d1b7b") : secret "dns-default-metrics-tls" not found
Sep 17 01:22:35.896814 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.896787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123f13ef-59c4-4635-8550-ee114b9d1b7b-tmp-dir\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.896926 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.896911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/123f13ef-59c4-4635-8550-ee114b9d1b7b-config-volume\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.898661 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.898643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7e25797-9a29-4a55-b0d9-bb5b89699bae-cert\") pod \"ingress-canary-pdvgq\" (UID: \"c7e25797-9a29-4a55-b0d9-bb5b89699bae\") " pod="openshift-ingress-canary/ingress-canary-pdvgq"
Sep 17 01:22:35.906492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.906474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdbc\" (UniqueName: \"kubernetes.io/projected/c7e25797-9a29-4a55-b0d9-bb5b89699bae-kube-api-access-tpdbc\") pod \"ingress-canary-pdvgq\" (UID: \"c7e25797-9a29-4a55-b0d9-bb5b89699bae\") " pod="openshift-ingress-canary/ingress-canary-pdvgq"
Sep 17 01:22:35.906585 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.906502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff7ct\" (UniqueName: \"kubernetes.io/projected/123f13ef-59c4-4635-8550-ee114b9d1b7b-kube-api-access-ff7ct\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:35.914886 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.914869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:22:35.917015 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.916995    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 01:22:35.917232 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.917217    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k6hcd\""
Sep 17 01:22:35.947521 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:35.947503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pdvgq"
Sep 17 01:22:36.231816 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:36.231553    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pdvgq"]
Sep 17 01:22:36.236546 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:22:36.236508    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e25797_9a29_4a55_b0d9_bb5b89699bae.slice/crio-972d117e45cbc0dca7ecf18c0cd7b31647eba77ddc42e2281edd47249fd0646f WatchSource:0}: Error finding container 972d117e45cbc0dca7ecf18c0cd7b31647eba77ddc42e2281edd47249fd0646f: Status 404 returned error can't find the container with id 972d117e45cbc0dca7ecf18c0cd7b31647eba77ddc42e2281edd47249fd0646f
Sep 17 01:22:36.301466 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:36.301432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:36.301622 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:36.301594    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 01:22:36.301622 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:36.301614    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c: secret "image-registry-tls" not found
Sep 17 01:22:36.301718 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:36.301679    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls podName:4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:37.301658412 +0000 UTC m=+35.972174533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls") pod "image-registry-7b4c7cb8b4-tfr4c" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a") : secret "image-registry-tls" not found
Sep 17 01:22:36.402066 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:36.402027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:36.402234 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:36.402200    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 01:22:36.402286 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:36.402271    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls podName:123f13ef-59c4-4635-8550-ee114b9d1b7b nodeName:}" failed. No retries permitted until 2026-09-17 01:22:37.402255213 +0000 UTC m=+36.072771280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls") pod "dns-default-nrzkz" (UID: "123f13ef-59c4-4635-8550-ee114b9d1b7b") : secret "dns-default-metrics-tls" not found
Sep 17 01:22:36.914369 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:36.914330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:22:36.914905 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:36.914340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:22:36.917259 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:36.917204    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 01:22:36.918223 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:36.918049    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 01:22:36.918223 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:36.918075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wvztc\""
Sep 17 01:22:36.918223 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:36.918122    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 01:22:37.154562 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:37.154510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pdvgq" event={"ID":"c7e25797-9a29-4a55-b0d9-bb5b89699bae","Type":"ContainerStarted","Data":"972d117e45cbc0dca7ecf18c0cd7b31647eba77ddc42e2281edd47249fd0646f"}
Sep 17 01:22:37.307649 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:37.307564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:37.307818 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:37.307720    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 01:22:37.307818 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:37.307734    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c: secret "image-registry-tls" not found
Sep 17 01:22:37.307818 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:37.307799    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls podName:4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:39.307780191 +0000 UTC m=+37.978296275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls") pod "image-registry-7b4c7cb8b4-tfr4c" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a") : secret "image-registry-tls" not found
Sep 17 01:22:37.408814 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:37.408778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:37.408998 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:37.408950    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 01:22:37.409065 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:37.409019    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls podName:123f13ef-59c4-4635-8550-ee114b9d1b7b nodeName:}" failed. No retries permitted until 2026-09-17 01:22:39.408999694 +0000 UTC m=+38.079515761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls") pod "dns-default-nrzkz" (UID: "123f13ef-59c4-4635-8550-ee114b9d1b7b") : secret "dns-default-metrics-tls" not found
Sep 17 01:22:39.320872 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:39.320840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:39.321373 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:39.321028    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 01:22:39.321373 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:39.321051    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c: secret "image-registry-tls" not found
Sep 17 01:22:39.321373 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:39.321131    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls podName:4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:43.32110891 +0000 UTC m=+41.991624983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls") pod "image-registry-7b4c7cb8b4-tfr4c" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a") : secret "image-registry-tls" not found
Sep 17 01:22:39.421926 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:39.421895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:39.422072 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:39.422047    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 01:22:39.422122 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:39.422111    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls podName:123f13ef-59c4-4635-8550-ee114b9d1b7b nodeName:}" failed. No retries permitted until 2026-09-17 01:22:43.422096108 +0000 UTC m=+42.092612175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls") pod "dns-default-nrzkz" (UID: "123f13ef-59c4-4635-8550-ee114b9d1b7b") : secret "dns-default-metrics-tls" not found
Sep 17 01:22:41.167549 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:41.167508    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c5faf65-3a14-47b8-8077-af57e33f8dfc" containerID="92f671f24f2cbc577bb1c04826eec32fd6283cb49b5737b14ad254f8eda6668b" exitCode=0
Sep 17 01:22:41.168347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:41.167564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvtgt" event={"ID":"7c5faf65-3a14-47b8-8077-af57e33f8dfc","Type":"ContainerDied","Data":"92f671f24f2cbc577bb1c04826eec32fd6283cb49b5737b14ad254f8eda6668b"}
Sep 17 01:22:41.169222 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:41.169195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pdvgq" event={"ID":"c7e25797-9a29-4a55-b0d9-bb5b89699bae","Type":"ContainerStarted","Data":"372bd5543c7675234875662df81b385cb198a6ca2a20282e4dadd972e59028be"}
Sep 17 01:22:41.201357 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:41.201310    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pdvgq" podStartSLOduration=2.336707585 podStartE2EDuration="6.201296605s" totalImagesPullingTime="3.86458902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:41.200667364 +0000 UTC m=+39.871183452" watchObservedRunningTime="2026-09-17 01:22:41.201296605 +0000 UTC m=+39.871812694"
Sep 17 01:22:42.174272 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:42.174240    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c5faf65-3a14-47b8-8077-af57e33f8dfc" containerID="91e9b318b29c005b99126e342004f2b3182a22babdef8b000ccb3c403574cbfe" exitCode=0
Sep 17 01:22:42.174726 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:42.174324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvtgt" event={"ID":"7c5faf65-3a14-47b8-8077-af57e33f8dfc","Type":"ContainerDied","Data":"91e9b318b29c005b99126e342004f2b3182a22babdef8b000ccb3c403574cbfe"}
Sep 17 01:22:43.179671 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:43.179641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvtgt" event={"ID":"7c5faf65-3a14-47b8-8077-af57e33f8dfc","Type":"ContainerStarted","Data":"d991a5d41e26d73d52adcd1c47c7ed07ad6af73562cdf31b060752e7dd18f19b"}
Sep 17 01:22:43.199356 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:43.199313    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hvtgt" podStartSLOduration=9.550194496 podStartE2EDuration="41.199296901s" totalImagesPullingTime="31.649102405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:43.19882024 +0000 UTC m=+41.869336328" watchObservedRunningTime="2026-09-17 01:22:43.199296901 +0000 UTC m=+41.869813013"
Sep 17 01:22:43.348350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:43.348311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:43.348498 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:43.348474    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 01:22:43.348498 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:43.348488    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c: secret "image-registry-tls" not found
Sep 17 01:22:43.348601 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:43.348562    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls podName:4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:51.348526914 +0000 UTC m=+50.019042999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls") pod "image-registry-7b4c7cb8b4-tfr4c" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a") : secret "image-registry-tls" not found
Sep 17 01:22:43.449341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:43.449260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:43.449476 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:43.449374    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 01:22:43.449476 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:43.449423    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls podName:123f13ef-59c4-4635-8550-ee114b9d1b7b nodeName:}" failed. No retries permitted until 2026-09-17 01:22:51.449410268 +0000 UTC m=+50.119926346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls") pod "dns-default-nrzkz" (UID: "123f13ef-59c4-4635-8550-ee114b9d1b7b") : secret "dns-default-metrics-tls" not found
Sep 17 01:22:51.403184 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:51.403143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:22:51.403613 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:51.403263    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 01:22:51.403613 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:51.403273    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c: secret "image-registry-tls" not found
Sep 17 01:22:51.403613 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:51.403332    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls podName:4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a nodeName:}" failed. No retries permitted until 2026-09-17 01:23:07.403319083 +0000 UTC m=+66.073835154 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls") pod "image-registry-7b4c7cb8b4-tfr4c" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a") : secret "image-registry-tls" not found
Sep 17 01:22:51.504496 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:22:51.504460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:22:51.504608 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:51.504588    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 01:22:51.504655 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:22:51.504646    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls podName:123f13ef-59c4-4635-8550-ee114b9d1b7b nodeName:}" failed. No retries permitted until 2026-09-17 01:23:07.504633194 +0000 UTC m=+66.175149274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls") pod "dns-default-nrzkz" (UID: "123f13ef-59c4-4635-8550-ee114b9d1b7b") : secret "dns-default-metrics-tls" not found
Sep 17 01:23:01.155635 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:01.155607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8kjk"
Sep 17 01:23:07.420005 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.419940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:23:07.420383 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:07.420071    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 01:23:07.420383 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:07.420082    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c: secret "image-registry-tls" not found
Sep 17 01:23:07.420383 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:07.420138    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls podName:4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a nodeName:}" failed. No retries permitted until 2026-09-17 01:23:39.420124464 +0000 UTC m=+98.090640531 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls") pod "image-registry-7b4c7cb8b4-tfr4c" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a") : secret "image-registry-tls" not found
Sep 17 01:23:07.520546 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.520496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:23:07.520710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.520589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:23:07.520710 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:07.520703    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 01:23:07.520819 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:07.520765    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls podName:123f13ef-59c4-4635-8550-ee114b9d1b7b nodeName:}" failed. No retries permitted until 2026-09-17 01:23:39.520746508 +0000 UTC m=+98.191262592 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls") pod "dns-default-nrzkz" (UID: "123f13ef-59c4-4635-8550-ee114b9d1b7b") : secret "dns-default-metrics-tls" not found
Sep 17 01:23:07.522901 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.522881    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 01:23:07.534548 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.534523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9850b1d-04e3-4bbd-9072-a9ac2dab518a-original-pull-secret\") pod \"global-pull-secret-syncer-mws4d\" (UID: \"a9850b1d-04e3-4bbd-9072-a9ac2dab518a\") " pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:23:07.535434 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.535418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mws4d"
Sep 17 01:23:07.621800 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.621759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:23:07.624048 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.624030    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 01:23:07.632796 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:07.632781    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 01:23:07.632872 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:07.632833    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs podName:ddb0ced7-abc3-439b-b250-ff90f01679cc nodeName:}" failed. No retries permitted until 2026-09-17 01:24:11.632816517 +0000 UTC m=+130.303332597 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs") pod "network-metrics-daemon-blh4d" (UID: "ddb0ced7-abc3-439b-b250-ff90f01679cc") : secret "metrics-daemon-secret" not found
Sep 17 01:23:07.722226 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.722151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486xp\" (UniqueName: \"kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp\") pod \"network-check-target-jsnp2\" (UID: \"1026d5e4-b4fd-4730-b074-7973a88b774c\") " pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:23:07.724672 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.724654    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 01:23:07.735325 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.735310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 01:23:07.746451 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.746426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-486xp\" (UniqueName: \"kubernetes.io/projected/1026d5e4-b4fd-4730-b074-7973a88b774c-kube-api-access-486xp\") pod \"network-check-target-jsnp2\" (UID: \"1026d5e4-b4fd-4730-b074-7973a88b774c\") " pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:23:07.797775 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.797752    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mws4d"]
Sep 17 01:23:07.800754 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:23:07.800723    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9850b1d_04e3_4bbd_9072_a9ac2dab518a.slice/crio-1f44561e35ad3e8e7149e9c6c4f968fa8b16157fa6770eeab7aa847ba4e12f82 WatchSource:0}: Error finding container 1f44561e35ad3e8e7149e9c6c4f968fa8b16157fa6770eeab7aa847ba4e12f82: Status 404 returned error can't find the container with id 1f44561e35ad3e8e7149e9c6c4f968fa8b16157fa6770eeab7aa847ba4e12f82
Sep 17 01:23:07.831245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.831224    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wvztc\""
Sep 17 01:23:07.838992 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:07.838974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:23:08.101791 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:08.101763    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jsnp2"]
Sep 17 01:23:08.104369 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:23:08.104345    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1026d5e4_b4fd_4730_b074_7973a88b774c.slice/crio-37c0cd8b2b0ea4b0565fd7e38e3f800d5cafdfc1ae51d7a322f92a349ef0f38a WatchSource:0}: Error finding container 37c0cd8b2b0ea4b0565fd7e38e3f800d5cafdfc1ae51d7a322f92a349ef0f38a: Status 404 returned error can't find the container with id 37c0cd8b2b0ea4b0565fd7e38e3f800d5cafdfc1ae51d7a322f92a349ef0f38a
Sep 17 01:23:08.237922 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:08.237889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jsnp2" event={"ID":"1026d5e4-b4fd-4730-b074-7973a88b774c","Type":"ContainerStarted","Data":"37c0cd8b2b0ea4b0565fd7e38e3f800d5cafdfc1ae51d7a322f92a349ef0f38a"}
Sep 17 01:23:08.239140 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:08.239113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mws4d" event={"ID":"a9850b1d-04e3-4bbd-9072-a9ac2dab518a","Type":"ContainerStarted","Data":"1f44561e35ad3e8e7149e9c6c4f968fa8b16157fa6770eeab7aa847ba4e12f82"}
Sep 17 01:23:13.253863 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:13.253825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jsnp2" event={"ID":"1026d5e4-b4fd-4730-b074-7973a88b774c","Type":"ContainerStarted","Data":"367bfbf461cc57745624ac2ae4ab895d76b635e695e1e2f397dc63b5e498c27d"}
Sep 17 01:23:13.254291 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:13.253885    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:23:13.255256 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:13.255231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mws4d" event={"ID":"a9850b1d-04e3-4bbd-9072-a9ac2dab518a","Type":"ContainerStarted","Data":"0ec9b2152360e1a45b7091cecdde89e30219d1bafea1cc3fcb0b6d11791190a4"}
Sep 17 01:23:13.269046 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:13.269008    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jsnp2" podStartSLOduration=66.789219621 podStartE2EDuration="1m11.268996255s" totalImagesPullingTime="4.479776634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:23:13.268480667 +0000 UTC m=+71.938996768" watchObservedRunningTime="2026-09-17 01:23:13.268996255 +0000 UTC m=+71.939512336"
Sep 17 01:23:39.436756 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:39.436714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:23:39.437128 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:39.436839    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 01:23:39.437128 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:39.436850    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c: secret "image-registry-tls" not found
Sep 17 01:23:39.437128 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:39.436902    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls podName:4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a nodeName:}" failed. No retries permitted until 2026-09-17 01:24:43.436887011 +0000 UTC m=+162.107403078 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls") pod "image-registry-7b4c7cb8b4-tfr4c" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a") : secret "image-registry-tls" not found
Sep 17 01:23:39.538050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:39.537970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:23:39.538133 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:39.537987    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 01:23:39.538168 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:23:39.538135    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls podName:123f13ef-59c4-4635-8550-ee114b9d1b7b nodeName:}" failed. No retries permitted until 2026-09-17 01:24:43.538117582 +0000 UTC m=+162.208633660 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls") pod "dns-default-nrzkz" (UID: "123f13ef-59c4-4635-8550-ee114b9d1b7b") : secret "dns-default-metrics-tls" not found
Sep 17 01:23:44.259990 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:44.259953    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jsnp2"
Sep 17 01:23:44.274036 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:23:44.273993    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mws4d" podStartSLOduration=97.486198949 podStartE2EDuration="1m42.273980756s" totalImagesPullingTime="4.787781807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:23:13.282227267 +0000 UTC m=+71.952743353" watchObservedRunningTime="2026-09-17 01:23:44.273980756 +0000 UTC m=+102.944496844"
Sep 17 01:24:11.656359 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:24:11.656312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:24:11.656903 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:24:11.656451    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 01:24:11.656903 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:24:11.656558    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs podName:ddb0ced7-abc3-439b-b250-ff90f01679cc nodeName:}" failed. No retries permitted until 2026-09-17 01:26:13.656511927 +0000 UTC m=+252.327027994 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs") pod "network-metrics-daemon-blh4d" (UID: "ddb0ced7-abc3-439b-b250-ff90f01679cc") : secret "metrics-daemon-secret" not found
Sep 17 01:24:38.614173 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:24:38.614128    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c" podUID="4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a"
Sep 17 01:24:38.641658 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:24:38.641629    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nrzkz" podUID="123f13ef-59c4-4635-8550-ee114b9d1b7b"
Sep 17 01:24:38.922900 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:24:38.922814    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:24:39.453490 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:24:39.453413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:24:39.453646 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:24:39.453414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:24:43.459878 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:24:43.459843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") pod \"image-registry-7b4c7cb8b4-tfr4c\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") " pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:24:43.460270 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:24:43.459971    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 01:24:43.460270 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:24:43.459983    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c: secret "image-registry-tls" not found
Sep 17 01:24:43.460270 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:24:43.460049    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls podName:4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a nodeName:}" failed. No retries permitted until 2026-09-17 01:26:45.460029604 +0000 UTC m=+284.130545673 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls") pod "image-registry-7b4c7cb8b4-tfr4c" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a") : secret "image-registry-tls" not found
Sep 17 01:24:43.560915 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:24:43.560874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:24:43.561080 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:24:43.561003    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 01:24:43.561080 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:24:43.561059    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls podName:123f13ef-59c4-4635-8550-ee114b9d1b7b nodeName:}" failed. No retries permitted until 2026-09-17 01:26:45.561046631 +0000 UTC m=+284.231562698 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls") pod "dns-default-nrzkz" (UID: "123f13ef-59c4-4635-8550-ee114b9d1b7b") : secret "dns-default-metrics-tls" not found
Sep 17 01:24:51.915142 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:24:51.915113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:26:04.770446 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.770410    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"]
Sep 17 01:26:04.772248 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.772231    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-qhwv5"]
Sep 17 01:26:04.772378 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.772357    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:04.773996 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.773979    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:04.775241 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.775222    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-72q5f\""
Sep 17 01:26:04.775352 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.775318    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 01:26:04.775352 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.775336    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 01:26:04.775458 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.775347    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 01:26:04.777207 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.777187    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 01:26:04.777207 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.777212    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 01:26:04.777505 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.777488    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 01:26:04.777600 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.777522    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-n8cfp\""
Sep 17 01:26:04.777600 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.777547    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 01:26:04.781107 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.781085    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 01:26:04.798657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.798638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-qhwv5"]
Sep 17 01:26:04.799288 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.799272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"]
Sep 17 01:26:04.862809 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.862787    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"]
Sep 17 01:26:04.865044 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.865023    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:04.868420 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.868394    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 01:26:04.868710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.868436    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 01:26:04.868710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.868511    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 01:26:04.868710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.868683    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 01:26:04.868969 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.868948    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7rw2b\""
Sep 17 01:26:04.869809 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.869788    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"]
Sep 17 01:26:04.871501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.871480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:04.874038 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.873989    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 01:26:04.874144 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.873989    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 01:26:04.874144 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.874068    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 01:26:04.874301 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.874274    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6nsmr\""
Sep 17 01:26:04.874691 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.874673    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 01:26:04.875922 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.875902    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"]
Sep 17 01:26:04.882207 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.882185    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"]
Sep 17 01:26:04.898766 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.898744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eae6b120-681a-4868-b818-151648b29629-snapshots\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:04.898886 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.898771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae6b120-681a-4868-b818-151648b29629-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:04.898886 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.898815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae6b120-681a-4868-b818-151648b29629-service-ca-bundle\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:04.898886 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.898868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbdjq\" (UniqueName: \"kubernetes.io/projected/b4fec28d-b231-40ec-a190-2e4a2c407f2a-kube-api-access-zbdjq\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:04.899018 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.898896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:04.899018 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.898917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eae6b120-681a-4868-b818-151648b29629-tmp\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:04.899018 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.898933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddx5\" (UniqueName: \"kubernetes.io/projected/eae6b120-681a-4868-b818-151648b29629-kube-api-access-nddx5\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:04.899018 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.898951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eae6b120-681a-4868-b818-151648b29629-serving-cert\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:04.999875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.999838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdjq\" (UniqueName: \"kubernetes.io/projected/b4fec28d-b231-40ec-a190-2e4a2c407f2a-kube-api-access-zbdjq\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:04.999875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.999874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a43fe2d1-5671-489e-803e-6f3920eb9cea-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:05.000109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.999895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:05.000109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.999913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eae6b120-681a-4868-b818-151648b29629-tmp\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.000109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.999952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddx5\" (UniqueName: \"kubernetes.io/projected/eae6b120-681a-4868-b818-151648b29629-kube-api-access-nddx5\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.000109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:04.999995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eae6b120-681a-4868-b818-151648b29629-serving-cert\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.000109 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:05.000018    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 01:26:05.000109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qc6n\" (UniqueName: \"kubernetes.io/projected/3fb80bee-2f30-457b-9263-6b8a93cf9c4d-kube-api-access-5qc6n\") pod \"service-ca-operator-cf7c776b7-vv2l2\" (UID: \"3fb80bee-2f30-457b-9263-6b8a93cf9c4d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:05.000109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:05.000109 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:05.000086    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls podName:b4fec28d-b231-40ec-a190-2e4a2c407f2a nodeName:}" failed. No retries permitted until 2026-09-17 01:26:05.500067706 +0000 UTC m=+244.170583810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8ghsb" (UID: "b4fec28d-b231-40ec-a190-2e4a2c407f2a") : secret "samples-operator-tls" not found
Sep 17 01:26:05.000501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3fb80bee-2f30-457b-9263-6b8a93cf9c4d-config\") pod \"service-ca-operator-cf7c776b7-vv2l2\" (UID: \"3fb80bee-2f30-457b-9263-6b8a93cf9c4d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:05.000501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6j66\" (UniqueName: \"kubernetes.io/projected/a43fe2d1-5671-489e-803e-6f3920eb9cea-kube-api-access-h6j66\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:05.000501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eae6b120-681a-4868-b818-151648b29629-snapshots\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.000501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3fb80bee-2f30-457b-9263-6b8a93cf9c4d-serving-cert\") pod \"service-ca-operator-cf7c776b7-vv2l2\" (UID: \"3fb80bee-2f30-457b-9263-6b8a93cf9c4d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:05.000501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae6b120-681a-4868-b818-151648b29629-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.000501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae6b120-681a-4868-b818-151648b29629-service-ca-bundle\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.000501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eae6b120-681a-4868-b818-151648b29629-tmp\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.000830 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eae6b120-681a-4868-b818-151648b29629-snapshots\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.000887 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae6b120-681a-4868-b818-151648b29629-service-ca-bundle\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.000982 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.000964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae6b120-681a-4868-b818-151648b29629-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.002569 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.002549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eae6b120-681a-4868-b818-151648b29629-serving-cert\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.009912 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.009893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdjq\" (UniqueName: \"kubernetes.io/projected/b4fec28d-b231-40ec-a190-2e4a2c407f2a-kube-api-access-zbdjq\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:05.010135 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.010113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddx5\" (UniqueName: \"kubernetes.io/projected/eae6b120-681a-4868-b818-151648b29629-kube-api-access-nddx5\") pod \"insights-operator-5844b856db-qhwv5\" (UID: \"eae6b120-681a-4868-b818-151648b29629\") " pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.087465 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.087446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-qhwv5"
Sep 17 01:26:05.100731 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.100702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a43fe2d1-5671-489e-803e-6f3920eb9cea-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:05.100836 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.100744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qc6n\" (UniqueName: \"kubernetes.io/projected/3fb80bee-2f30-457b-9263-6b8a93cf9c4d-kube-api-access-5qc6n\") pod \"service-ca-operator-cf7c776b7-vv2l2\" (UID: \"3fb80bee-2f30-457b-9263-6b8a93cf9c4d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:05.100898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.100857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:05.100898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.100886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3fb80bee-2f30-457b-9263-6b8a93cf9c4d-config\") pod \"service-ca-operator-cf7c776b7-vv2l2\" (UID: \"3fb80bee-2f30-457b-9263-6b8a93cf9c4d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:05.101000 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.100905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6j66\" (UniqueName: \"kubernetes.io/projected/a43fe2d1-5671-489e-803e-6f3920eb9cea-kube-api-access-h6j66\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:05.101000 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.100936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3fb80bee-2f30-457b-9263-6b8a93cf9c4d-serving-cert\") pod \"service-ca-operator-cf7c776b7-vv2l2\" (UID: \"3fb80bee-2f30-457b-9263-6b8a93cf9c4d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:05.101000 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:05.100980    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 01:26:05.101150 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:05.101058    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls podName:a43fe2d1-5671-489e-803e-6f3920eb9cea nodeName:}" failed. No retries permitted until 2026-09-17 01:26:05.601039369 +0000 UTC m=+244.271555450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ffmtx" (UID: "a43fe2d1-5671-489e-803e-6f3920eb9cea") : secret "cluster-monitoring-operator-tls" not found
Sep 17 01:26:05.101378 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.101360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a43fe2d1-5671-489e-803e-6f3920eb9cea-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:05.101487 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.101469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3fb80bee-2f30-457b-9263-6b8a93cf9c4d-config\") pod \"service-ca-operator-cf7c776b7-vv2l2\" (UID: \"3fb80bee-2f30-457b-9263-6b8a93cf9c4d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:05.102935 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.102914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3fb80bee-2f30-457b-9263-6b8a93cf9c4d-serving-cert\") pod \"service-ca-operator-cf7c776b7-vv2l2\" (UID: \"3fb80bee-2f30-457b-9263-6b8a93cf9c4d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:05.108899 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.108879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qc6n\" (UniqueName: \"kubernetes.io/projected/3fb80bee-2f30-457b-9263-6b8a93cf9c4d-kube-api-access-5qc6n\") pod \"service-ca-operator-cf7c776b7-vv2l2\" (UID: \"3fb80bee-2f30-457b-9263-6b8a93cf9c4d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:05.108999 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.108932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6j66\" (UniqueName: \"kubernetes.io/projected/a43fe2d1-5671-489e-803e-6f3920eb9cea-kube-api-access-h6j66\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:05.174313 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.174284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"
Sep 17 01:26:05.352205 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.352118    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-qhwv5"]
Sep 17 01:26:05.355055 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:05.355021    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae6b120_681a_4868_b818_151648b29629.slice/crio-a53c8a64df1d3ffb1e33bbc703a6992311e614e6e77dba792d3773d38f7cefa9 WatchSource:0}: Error finding container a53c8a64df1d3ffb1e33bbc703a6992311e614e6e77dba792d3773d38f7cefa9: Status 404 returned error can't find the container with id a53c8a64df1d3ffb1e33bbc703a6992311e614e6e77dba792d3773d38f7cefa9
Sep 17 01:26:05.434757 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.434733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2"]
Sep 17 01:26:05.437159 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:05.437138    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb80bee_2f30_457b_9263_6b8a93cf9c4d.slice/crio-92ceb6f4ad01670332b8729adffe5838a336515573bcf8d0dc0a7497a455f8d5 WatchSource:0}: Error finding container 92ceb6f4ad01670332b8729adffe5838a336515573bcf8d0dc0a7497a455f8d5: Status 404 returned error can't find the container with id 92ceb6f4ad01670332b8729adffe5838a336515573bcf8d0dc0a7497a455f8d5
Sep 17 01:26:05.504501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.504473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:05.504669 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:05.504615    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 01:26:05.504732 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:05.504682    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls podName:b4fec28d-b231-40ec-a190-2e4a2c407f2a nodeName:}" failed. No retries permitted until 2026-09-17 01:26:06.504662913 +0000 UTC m=+245.175178983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8ghsb" (UID: "b4fec28d-b231-40ec-a190-2e4a2c407f2a") : secret "samples-operator-tls" not found
Sep 17 01:26:05.605295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.605220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:05.605419 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:05.605365    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 01:26:05.605466 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:05.605429    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls podName:a43fe2d1-5671-489e-803e-6f3920eb9cea nodeName:}" failed. No retries permitted until 2026-09-17 01:26:06.605410225 +0000 UTC m=+245.275926307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ffmtx" (UID: "a43fe2d1-5671-489e-803e-6f3920eb9cea") : secret "cluster-monitoring-operator-tls" not found
Sep 17 01:26:05.653943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.653911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2" event={"ID":"3fb80bee-2f30-457b-9263-6b8a93cf9c4d","Type":"ContainerStarted","Data":"92ceb6f4ad01670332b8729adffe5838a336515573bcf8d0dc0a7497a455f8d5"}
Sep 17 01:26:05.654957 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:05.654935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qhwv5" event={"ID":"eae6b120-681a-4868-b818-151648b29629","Type":"ContainerStarted","Data":"a53c8a64df1d3ffb1e33bbc703a6992311e614e6e77dba792d3773d38f7cefa9"}
Sep 17 01:26:06.511093 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:06.510967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:06.511587 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:06.511175    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 01:26:06.511587 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:06.511249    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls podName:b4fec28d-b231-40ec-a190-2e4a2c407f2a nodeName:}" failed. No retries permitted until 2026-09-17 01:26:08.511227922 +0000 UTC m=+247.181744010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8ghsb" (UID: "b4fec28d-b231-40ec-a190-2e4a2c407f2a") : secret "samples-operator-tls" not found
Sep 17 01:26:06.611864 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:06.611824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:06.612032 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:06.611994    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 01:26:06.612087 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:06.612074    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls podName:a43fe2d1-5671-489e-803e-6f3920eb9cea nodeName:}" failed. No retries permitted until 2026-09-17 01:26:08.6120537 +0000 UTC m=+247.282569780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ffmtx" (UID: "a43fe2d1-5671-489e-803e-6f3920eb9cea") : secret "cluster-monitoring-operator-tls" not found
Sep 17 01:26:08.526720 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.526680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:08.527153 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:08.526870    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 01:26:08.527153 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:08.526941    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls podName:b4fec28d-b231-40ec-a190-2e4a2c407f2a nodeName:}" failed. No retries permitted until 2026-09-17 01:26:12.526919846 +0000 UTC m=+251.197435914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8ghsb" (UID: "b4fec28d-b231-40ec-a190-2e4a2c407f2a") : secret "samples-operator-tls" not found
Sep 17 01:26:08.627611 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.627568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:08.627769 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:08.627705    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 01:26:08.627825 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:08.627776    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls podName:a43fe2d1-5671-489e-803e-6f3920eb9cea nodeName:}" failed. No retries permitted until 2026-09-17 01:26:12.627760366 +0000 UTC m=+251.298276463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ffmtx" (UID: "a43fe2d1-5671-489e-803e-6f3920eb9cea") : secret "cluster-monitoring-operator-tls" not found
Sep 17 01:26:08.665185 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.665151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2" event={"ID":"3fb80bee-2f30-457b-9263-6b8a93cf9c4d","Type":"ContainerStarted","Data":"1c66378b2d97f26fc2d3d250b77dee35162d6e50175cc8361a89eee1e017cccc"}
Sep 17 01:26:08.666743 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.666718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qhwv5" event={"ID":"eae6b120-681a-4868-b818-151648b29629","Type":"ContainerStarted","Data":"180fe54afab6225409c872b58ee5a3ddb9cc4a6b065a8eeee609dde03e8bfb4f"}
Sep 17 01:26:08.679469 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.679423    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2" podStartSLOduration=2.368537631 podStartE2EDuration="4.679409985s" totalImagesPullingTime="2.310872354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:08.678438838 +0000 UTC m=+247.348954946" watchObservedRunningTime="2026-09-17 01:26:08.679409985 +0000 UTC m=+247.349926066"
Sep 17 01:26:08.693465 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.693414    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-qhwv5" podStartSLOduration=2.303234867 podStartE2EDuration="4.693396942s" totalImagesPullingTime="2.390162075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:08.692266823 +0000 UTC m=+247.362782911" watchObservedRunningTime="2026-09-17 01:26:08.693396942 +0000 UTC m=+247.363913032"
Sep 17 01:26:08.797395 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.797366    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-twqpd"]
Sep 17 01:26:08.799529 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.799507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:08.801955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.801931    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 01:26:08.801955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.801931    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 01:26:08.802128 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.801987    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qkzkb\""
Sep 17 01:26:08.808157 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.808126    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-twqpd"]
Sep 17 01:26:08.931155 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.931120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:08.931305 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.931179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4fb7434b-5484-4868-a9e2-78f64165b632-crio-socket\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:08.931347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.931303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:08.931419 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.931398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fb7434b-5484-4868-a9e2-78f64165b632-data-volume\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:08.931458 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:08.931427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-899wg\" (UniqueName: \"kubernetes.io/projected/4fb7434b-5484-4868-a9e2-78f64165b632-kube-api-access-899wg\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.032317 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:09.032281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fb7434b-5484-4868-a9e2-78f64165b632-data-volume\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.032317 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:09.032313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-899wg\" (UniqueName: \"kubernetes.io/projected/4fb7434b-5484-4868-a9e2-78f64165b632-kube-api-access-899wg\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.032521 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:09.032337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.032521 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:09.032371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4fb7434b-5484-4868-a9e2-78f64165b632-crio-socket\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.032521 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:09.032415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.032521 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:09.032474    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 01:26:09.032521 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:09.032518    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 01:26:09.032737 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:09.032554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4fb7434b-5484-4868-a9e2-78f64165b632-crio-socket\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.032737 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:09.032575    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls podName:4fb7434b-5484-4868-a9e2-78f64165b632 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:09.532555564 +0000 UTC m=+248.203071636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-twqpd" (UID: "4fb7434b-5484-4868-a9e2-78f64165b632") : secret "insights-runtime-extractor-tls" not found
Sep 17 01:26:09.032737 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:09.032597    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm podName:4fb7434b-5484-4868-a9e2-78f64165b632 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:09.532586714 +0000 UTC m=+248.203102786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm") pod "insights-runtime-extractor-twqpd" (UID: "4fb7434b-5484-4868-a9e2-78f64165b632") : configmap "kube-rbac-proxy" not found
Sep 17 01:26:09.032737 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:09.032719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fb7434b-5484-4868-a9e2-78f64165b632-data-volume\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.042173 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:09.042144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-899wg\" (UniqueName: \"kubernetes.io/projected/4fb7434b-5484-4868-a9e2-78f64165b632-kube-api-access-899wg\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.537745 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:09.537712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.538121 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:09.537781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:09.538121 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:09.537869    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 01:26:09.538121 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:09.537872    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 01:26:09.538121 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:09.537932    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls podName:4fb7434b-5484-4868-a9e2-78f64165b632 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:10.537916968 +0000 UTC m=+249.208433035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-twqpd" (UID: "4fb7434b-5484-4868-a9e2-78f64165b632") : secret "insights-runtime-extractor-tls" not found
Sep 17 01:26:09.538121 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:09.537947    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm podName:4fb7434b-5484-4868-a9e2-78f64165b632 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:10.537941053 +0000 UTC m=+249.208457121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm") pod "insights-runtime-extractor-twqpd" (UID: "4fb7434b-5484-4868-a9e2-78f64165b632") : configmap "kube-rbac-proxy" not found
Sep 17 01:26:10.545192 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.545157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:10.545767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.545229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:10.545767 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:10.545292    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 01:26:10.545767 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:10.545329    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 01:26:10.545767 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:10.545347    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm podName:4fb7434b-5484-4868-a9e2-78f64165b632 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:12.545332875 +0000 UTC m=+251.215848943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm") pod "insights-runtime-extractor-twqpd" (UID: "4fb7434b-5484-4868-a9e2-78f64165b632") : configmap "kube-rbac-proxy" not found
Sep 17 01:26:10.545767 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:10.545408    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls podName:4fb7434b-5484-4868-a9e2-78f64165b632 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:12.545389984 +0000 UTC m=+251.215906053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-twqpd" (UID: "4fb7434b-5484-4868-a9e2-78f64165b632") : secret "insights-runtime-extractor-tls" not found
Sep 17 01:26:10.748863 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.748837    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mxzkv_d2d5b623-d9d2-4621-bc65-66241a002022/dns-node-resolver/0.log"
Sep 17 01:26:10.978677 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.978601    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-r7ftl"]
Sep 17 01:26:10.981255 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.981235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:10.983583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.983564    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 01:26:10.983583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.983572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lt78m\""
Sep 17 01:26:10.984525 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.984508    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 01:26:10.984778 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.984764    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 01:26:10.984855 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.984776    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 01:26:10.988208 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:10.988182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-r7ftl"]
Sep 17 01:26:11.048986 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.048961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/57141473-0b78-42b9-81ec-67822e3a7746-signing-key\") pod \"service-ca-568869fdc5-r7ftl\" (UID: \"57141473-0b78-42b9-81ec-67822e3a7746\") " pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:11.049095 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.049039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/57141473-0b78-42b9-81ec-67822e3a7746-signing-cabundle\") pod \"service-ca-568869fdc5-r7ftl\" (UID: \"57141473-0b78-42b9-81ec-67822e3a7746\") " pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:11.049095 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.049065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j22nb\" (UniqueName: \"kubernetes.io/projected/57141473-0b78-42b9-81ec-67822e3a7746-kube-api-access-j22nb\") pod \"service-ca-568869fdc5-r7ftl\" (UID: \"57141473-0b78-42b9-81ec-67822e3a7746\") " pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:11.149469 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.149445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j22nb\" (UniqueName: \"kubernetes.io/projected/57141473-0b78-42b9-81ec-67822e3a7746-kube-api-access-j22nb\") pod \"service-ca-568869fdc5-r7ftl\" (UID: \"57141473-0b78-42b9-81ec-67822e3a7746\") " pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:11.149594 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.149514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/57141473-0b78-42b9-81ec-67822e3a7746-signing-key\") pod \"service-ca-568869fdc5-r7ftl\" (UID: \"57141473-0b78-42b9-81ec-67822e3a7746\") " pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:11.149640 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.149618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/57141473-0b78-42b9-81ec-67822e3a7746-signing-cabundle\") pod \"service-ca-568869fdc5-r7ftl\" (UID: \"57141473-0b78-42b9-81ec-67822e3a7746\") " pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:11.150337 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.150317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/57141473-0b78-42b9-81ec-67822e3a7746-signing-cabundle\") pod \"service-ca-568869fdc5-r7ftl\" (UID: \"57141473-0b78-42b9-81ec-67822e3a7746\") " pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:11.151913 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.151895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/57141473-0b78-42b9-81ec-67822e3a7746-signing-key\") pod \"service-ca-568869fdc5-r7ftl\" (UID: \"57141473-0b78-42b9-81ec-67822e3a7746\") " pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:11.156981 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.156954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j22nb\" (UniqueName: \"kubernetes.io/projected/57141473-0b78-42b9-81ec-67822e3a7746-kube-api-access-j22nb\") pod \"service-ca-568869fdc5-r7ftl\" (UID: \"57141473-0b78-42b9-81ec-67822e3a7746\") " pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:11.290482 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.290391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-r7ftl"
Sep 17 01:26:11.551359 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.551338    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-22cxn_33989fb0-89ab-47a8-90a4-2fbd2332d884/node-ca/0.log"
Sep 17 01:26:11.559221 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.559198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-r7ftl"]
Sep 17 01:26:11.562046 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:11.562021    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57141473_0b78_42b9_81ec_67822e3a7746.slice/crio-2b208c4aa8bf9f369e320f5b2e8951f88767f9e4ca23676e875fa28168dd11db WatchSource:0}: Error finding container 2b208c4aa8bf9f369e320f5b2e8951f88767f9e4ca23676e875fa28168dd11db: Status 404 returned error can't find the container with id 2b208c4aa8bf9f369e320f5b2e8951f88767f9e4ca23676e875fa28168dd11db
Sep 17 01:26:11.677900 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.677868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-r7ftl" event={"ID":"57141473-0b78-42b9-81ec-67822e3a7746","Type":"ContainerStarted","Data":"30b1cf5369829ca99519b1e8a003c882916bf039e7cfb95f2c65f85366c4af44"}
Sep 17 01:26:11.678059 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.677919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-r7ftl" event={"ID":"57141473-0b78-42b9-81ec-67822e3a7746","Type":"ContainerStarted","Data":"2b208c4aa8bf9f369e320f5b2e8951f88767f9e4ca23676e875fa28168dd11db"}
Sep 17 01:26:11.695530 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:11.695476    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-r7ftl" podStartSLOduration=1.695457104 podStartE2EDuration="1.695457104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:11.693458257 +0000 UTC m=+250.363974359" watchObservedRunningTime="2026-09-17 01:26:11.695457104 +0000 UTC m=+250.365973194"
Sep 17 01:26:12.559505 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:12.559466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:12.559955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:12.559573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:12.559955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:12.559599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:12.559955 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:12.559627    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 01:26:12.559955 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:12.559698    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm podName:4fb7434b-5484-4868-a9e2-78f64165b632 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:16.559681487 +0000 UTC m=+255.230197555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm") pod "insights-runtime-extractor-twqpd" (UID: "4fb7434b-5484-4868-a9e2-78f64165b632") : configmap "kube-rbac-proxy" not found
Sep 17 01:26:12.559955 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:12.559723    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 01:26:12.559955 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:12.559780    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls podName:4fb7434b-5484-4868-a9e2-78f64165b632 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:16.559762329 +0000 UTC m=+255.230278402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-twqpd" (UID: "4fb7434b-5484-4868-a9e2-78f64165b632") : secret "insights-runtime-extractor-tls" not found
Sep 17 01:26:12.559955 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:12.559723    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 01:26:12.559955 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:12.559823    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls podName:b4fec28d-b231-40ec-a190-2e4a2c407f2a nodeName:}" failed. No retries permitted until 2026-09-17 01:26:20.559813934 +0000 UTC m=+259.230330005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8ghsb" (UID: "b4fec28d-b231-40ec-a190-2e4a2c407f2a") : secret "samples-operator-tls" not found
Sep 17 01:26:12.660773 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:12.660739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:12.660938 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:12.660849    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 01:26:12.660938 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:12.660904    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls podName:a43fe2d1-5671-489e-803e-6f3920eb9cea nodeName:}" failed. No retries permitted until 2026-09-17 01:26:20.660890254 +0000 UTC m=+259.331406321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ffmtx" (UID: "a43fe2d1-5671-489e-803e-6f3920eb9cea") : secret "cluster-monitoring-operator-tls" not found
Sep 17 01:26:12.748505 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:12.748478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pdvgq_c7e25797-9a29-4a55-b0d9-bb5b89699bae/serve-healthcheck-canary/0.log"
Sep 17 01:26:13.671297 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:13.671262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:26:13.671806 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:13.671429    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 01:26:13.671806 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:13.671509    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs podName:ddb0ced7-abc3-439b-b250-ff90f01679cc nodeName:}" failed. No retries permitted until 2026-09-17 01:28:15.671488175 +0000 UTC m=+374.342004263 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs") pod "network-metrics-daemon-blh4d" (UID: "ddb0ced7-abc3-439b-b250-ff90f01679cc") : secret "metrics-daemon-secret" not found
Sep 17 01:26:16.594222 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:16.594184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:16.594620 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:16.594253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:16.594620 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:16.594341    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 01:26:16.594620 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:16.594365    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 01:26:16.594620 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:16.594393    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm podName:4fb7434b-5484-4868-a9e2-78f64165b632 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:24.594379432 +0000 UTC m=+263.264895499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm") pod "insights-runtime-extractor-twqpd" (UID: "4fb7434b-5484-4868-a9e2-78f64165b632") : configmap "kube-rbac-proxy" not found
Sep 17 01:26:16.594620 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:16.594428    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls podName:4fb7434b-5484-4868-a9e2-78f64165b632 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:24.594411079 +0000 UTC m=+263.264927160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-twqpd" (UID: "4fb7434b-5484-4868-a9e2-78f64165b632") : secret "insights-runtime-extractor-tls" not found
Sep 17 01:26:20.626742 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:20.626705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:20.629057 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:20.629034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4fec28d-b231-40ec-a190-2e4a2c407f2a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8ghsb\" (UID: \"b4fec28d-b231-40ec-a190-2e4a2c407f2a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:20.682104 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:20.682068    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"
Sep 17 01:26:20.728438 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:20.728320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:20.731761 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:20.731693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a43fe2d1-5671-489e-803e-6f3920eb9cea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ffmtx\" (UID: \"a43fe2d1-5671-489e-803e-6f3920eb9cea\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:20.781000 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:20.780967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"
Sep 17 01:26:20.950267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:20.950197    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb"]
Sep 17 01:26:21.048132 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:21.048101    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx"]
Sep 17 01:26:21.051066 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:21.051045    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43fe2d1_5671_489e_803e_6f3920eb9cea.slice/crio-d167657aefed072a2609a71f198f375052222d4696e57c5201816cf89a27e1ab WatchSource:0}: Error finding container d167657aefed072a2609a71f198f375052222d4696e57c5201816cf89a27e1ab: Status 404 returned error can't find the container with id d167657aefed072a2609a71f198f375052222d4696e57c5201816cf89a27e1ab
Sep 17 01:26:21.709575 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:21.709505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx" event={"ID":"a43fe2d1-5671-489e-803e-6f3920eb9cea","Type":"ContainerStarted","Data":"d167657aefed072a2609a71f198f375052222d4696e57c5201816cf89a27e1ab"}
Sep 17 01:26:21.710861 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:21.710837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb" event={"ID":"b4fec28d-b231-40ec-a190-2e4a2c407f2a","Type":"ContainerStarted","Data":"522c5b402276c4960b60f08b8c0f64a5fe461e08c945cc3ed07e6541a098609f"}
Sep 17 01:26:23.719986 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:23.719889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb" event={"ID":"b4fec28d-b231-40ec-a190-2e4a2c407f2a","Type":"ContainerStarted","Data":"860e074de63acc8e1b76528b44eedefff4bfbe5d3420606983cad703d9e3c096"}
Sep 17 01:26:23.720435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:23.719998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb" event={"ID":"b4fec28d-b231-40ec-a190-2e4a2c407f2a","Type":"ContainerStarted","Data":"3519aeec5532070769fb7a9b3360114e2fa359111481cabb57deb9394166bc63"}
Sep 17 01:26:23.721797 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:23.721767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx" event={"ID":"a43fe2d1-5671-489e-803e-6f3920eb9cea","Type":"ContainerStarted","Data":"85934fb31d13f5b0c17dcd9225474600d5d4bfa8ad62cca1e8c6801ac2f4b328"}
Sep 17 01:26:23.735970 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:23.735928    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8ghsb" podStartSLOduration=17.288506277 podStartE2EDuration="19.735914852s" totalImagesPullingTime="2.447408575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:23.734958745 +0000 UTC m=+262.405474835" watchObservedRunningTime="2026-09-17 01:26:23.735914852 +0000 UTC m=+262.406430941"
Sep 17 01:26:23.752907 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:23.752854    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ffmtx" podStartSLOduration=17.375342782 podStartE2EDuration="19.752837092s" totalImagesPullingTime="2.37749431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:23.752298479 +0000 UTC m=+262.422814570" watchObservedRunningTime="2026-09-17 01:26:23.752837092 +0000 UTC m=+262.423353183"
Sep 17 01:26:24.663010 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:24.662974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:24.663183 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:24.663034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:24.663511 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:24.663481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4fb7434b-5484-4868-a9e2-78f64165b632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:24.665236 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:24.665217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4fb7434b-5484-4868-a9e2-78f64165b632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twqpd\" (UID: \"4fb7434b-5484-4868-a9e2-78f64165b632\") " pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:24.708526 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:24.708492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-twqpd"
Sep 17 01:26:24.975901 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:24.975829    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-twqpd"]
Sep 17 01:26:24.978795 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:24.978770    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb7434b_5484_4868_a9e2_78f64165b632.slice/crio-8dee1d0c9bdd7a839910bf12574a531e6a97b71d5bd0b3db08044addb0516d16 WatchSource:0}: Error finding container 8dee1d0c9bdd7a839910bf12574a531e6a97b71d5bd0b3db08044addb0516d16: Status 404 returned error can't find the container with id 8dee1d0c9bdd7a839910bf12574a531e6a97b71d5bd0b3db08044addb0516d16
Sep 17 01:26:25.729864 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:25.729833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-twqpd" event={"ID":"4fb7434b-5484-4868-a9e2-78f64165b632","Type":"ContainerStarted","Data":"2451dcc5fc10cb95d2afcbe3fc11119e9fe3a4eb13208d345cde6b092d85c237"}
Sep 17 01:26:25.729997 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:25.729884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-twqpd" event={"ID":"4fb7434b-5484-4868-a9e2-78f64165b632","Type":"ContainerStarted","Data":"c7d13b7fa8c6d297810a25c358ca988fc5920687f517ed64d06359f2956038a4"}
Sep 17 01:26:25.729997 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:25.729902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-twqpd" event={"ID":"4fb7434b-5484-4868-a9e2-78f64165b632","Type":"ContainerStarted","Data":"8dee1d0c9bdd7a839910bf12574a531e6a97b71d5bd0b3db08044addb0516d16"}
Sep 17 01:26:27.738837 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:27.738800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-twqpd" event={"ID":"4fb7434b-5484-4868-a9e2-78f64165b632","Type":"ContainerStarted","Data":"ffab3376a40531b0626a609eb76511effda7c516d8a6682a3b1c3d5a0eabb880"}
Sep 17 01:26:27.757832 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:27.757788    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-twqpd" podStartSLOduration=17.693363088 podStartE2EDuration="19.757772047s" totalImagesPullingTime="2.064408959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:26:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:27.757622631 +0000 UTC m=+266.428138720" watchObservedRunningTime="2026-09-17 01:26:27.757772047 +0000 UTC m=+266.428288137"
Sep 17 01:26:30.966543 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:30.966513    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"]
Sep 17 01:26:30.966889 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:30.966699    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c" podUID="4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a"
Sep 17 01:26:31.001314 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.001285    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f8798949b-9v5sv"]
Sep 17 01:26:31.004411 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.004396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.010131 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.010035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/975e24fc-f293-4864-98b3-caef33a0e4ec-trusted-ca\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.010131 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.010076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/975e24fc-f293-4864-98b3-caef33a0e4ec-registry-tls\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.010131 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.010127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/975e24fc-f293-4864-98b3-caef33a0e4ec-installation-pull-secrets\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.010291 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.010151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4wc\" (UniqueName: \"kubernetes.io/projected/975e24fc-f293-4864-98b3-caef33a0e4ec-kube-api-access-kt4wc\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.010291 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.010190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/975e24fc-f293-4864-98b3-caef33a0e4ec-ca-trust-extracted\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.010291 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.010217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/975e24fc-f293-4864-98b3-caef33a0e4ec-bound-sa-token\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.010291 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.010238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/975e24fc-f293-4864-98b3-caef33a0e4ec-registry-certificates\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.010291 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.010256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/975e24fc-f293-4864-98b3-caef33a0e4ec-image-registry-private-configuration\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.020021 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.019999    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f8798949b-9v5sv"]
Sep 17 01:26:31.110580 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.110553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/975e24fc-f293-4864-98b3-caef33a0e4ec-trusted-ca\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.110721 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.110586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/975e24fc-f293-4864-98b3-caef33a0e4ec-registry-tls\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.110721 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.110615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/975e24fc-f293-4864-98b3-caef33a0e4ec-installation-pull-secrets\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.110721 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.110635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4wc\" (UniqueName: \"kubernetes.io/projected/975e24fc-f293-4864-98b3-caef33a0e4ec-kube-api-access-kt4wc\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.110721 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.110677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/975e24fc-f293-4864-98b3-caef33a0e4ec-ca-trust-extracted\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.110721 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.110706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/975e24fc-f293-4864-98b3-caef33a0e4ec-bound-sa-token\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.110996 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.110732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/975e24fc-f293-4864-98b3-caef33a0e4ec-registry-certificates\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.110996 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.110784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/975e24fc-f293-4864-98b3-caef33a0e4ec-image-registry-private-configuration\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.111326 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.111296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/975e24fc-f293-4864-98b3-caef33a0e4ec-ca-trust-extracted\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.111477 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.111460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/975e24fc-f293-4864-98b3-caef33a0e4ec-registry-certificates\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.111636 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.111614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/975e24fc-f293-4864-98b3-caef33a0e4ec-trusted-ca\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.113239 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.113214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/975e24fc-f293-4864-98b3-caef33a0e4ec-registry-tls\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.113320 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.113287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/975e24fc-f293-4864-98b3-caef33a0e4ec-installation-pull-secrets\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.113320 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.113298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/975e24fc-f293-4864-98b3-caef33a0e4ec-image-registry-private-configuration\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.119100 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.119072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4wc\" (UniqueName: \"kubernetes.io/projected/975e24fc-f293-4864-98b3-caef33a0e4ec-kube-api-access-kt4wc\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.119211 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.119156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/975e24fc-f293-4864-98b3-caef33a0e4ec-bound-sa-token\") pod \"image-registry-7f8798949b-9v5sv\" (UID: \"975e24fc-f293-4864-98b3-caef33a0e4ec\") " pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.314209 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.314183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dn79c\""
Sep 17 01:26:31.322941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.322918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.596807 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.596730    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f8798949b-9v5sv"]
Sep 17 01:26:31.600431 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:31.600399    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975e24fc_f293_4864_98b3_caef33a0e4ec.slice/crio-69d2fb794808e30a5f3307942b0edc0851d76decbe58d630d922bf6a8ab5b2f8 WatchSource:0}: Error finding container 69d2fb794808e30a5f3307942b0edc0851d76decbe58d630d922bf6a8ab5b2f8: Status 404 returned error can't find the container with id 69d2fb794808e30a5f3307942b0edc0851d76decbe58d630d922bf6a8ab5b2f8
Sep 17 01:26:31.754313 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.754286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:26:31.754507 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.754316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f8798949b-9v5sv" event={"ID":"975e24fc-f293-4864-98b3-caef33a0e4ec","Type":"ContainerStarted","Data":"244415dd4b7e37141748e0d3d19e6e1feeeaa03640ee79704b5946958f02cb07"}
Sep 17 01:26:31.754507 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.754376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f8798949b-9v5sv" event={"ID":"975e24fc-f293-4864-98b3-caef33a0e4ec","Type":"ContainerStarted","Data":"69d2fb794808e30a5f3307942b0edc0851d76decbe58d630d922bf6a8ab5b2f8"}
Sep 17 01:26:31.754507 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.754402    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:31.761821 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.761802    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:26:31.787659 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.787609    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f8798949b-9v5sv" podStartSLOduration=1.787590738 podStartE2EDuration="1.787590738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:31.77177196 +0000 UTC m=+270.442288050" watchObservedRunningTime="2026-09-17 01:26:31.787590738 +0000 UTC m=+270.458106828"
Sep 17 01:26:31.815067 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.815040    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-bound-sa-token\") pod \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") "
Sep 17 01:26:31.815262 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.815083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-certificates\") pod \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") "
Sep 17 01:26:31.815262 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.815101    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-installation-pull-secrets\") pod \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") "
Sep 17 01:26:31.815262 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.815126    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-trusted-ca\") pod \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") "
Sep 17 01:26:31.815262 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.815199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxzgv\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-kube-api-access-dxzgv\") pod \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") "
Sep 17 01:26:31.815262 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.815230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-image-registry-private-configuration\") pod \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") "
Sep 17 01:26:31.815512 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.815315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-ca-trust-extracted\") pod \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\" (UID: \"4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a\") "
Sep 17 01:26:31.815603 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.815510    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:26:31.815662 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.815622    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:26:31.816416 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.816206    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:26:31.816416 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.816289    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-certificates\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:26:31.816416 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.816309    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-trusted-ca\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:26:31.816416 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.816325    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-ca-trust-extracted\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:26:31.817952 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.817929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:26:31.818055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.817998    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:26:31.818055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.818034    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-kube-api-access-dxzgv" (OuterVolumeSpecName: "kube-api-access-dxzgv") pod "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a"). InnerVolumeSpecName "kube-api-access-dxzgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:26:31.818614 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.818591    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a" (UID: "4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:26:31.916808 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.916733    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-bound-sa-token\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:26:31.916808 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.916757    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-installation-pull-secrets\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:26:31.916808 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.916766    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxzgv\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-kube-api-access-dxzgv\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:26:31.916808 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:31.916778    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-image-registry-private-configuration\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:26:32.757767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:32.757731    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"
Sep 17 01:26:32.786087 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:32.786057    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"]
Sep 17 01:26:32.789424 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:32.789402    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b4c7cb8b4-tfr4c"]
Sep 17 01:26:32.824050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:32.824023    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a-registry-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:26:33.918509 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:33.918475    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a" path="/var/lib/kubelet/pods/4a7be421-e3da-4d66-bb6a-2b0f2b2ac94a/volumes"
Sep 17 01:26:33.973107 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:33.973076    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"]
Sep 17 01:26:33.977692 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:33.977670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:33.980710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:33.980685    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5f9v4\""
Sep 17 01:26:33.980821 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:33.980742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 01:26:33.980821 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:33.980742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 01:26:33.980943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:33.980761    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 01:26:33.984094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:33.984013    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"]
Sep 17 01:26:34.033091 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.033060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b3cb156-e78f-45e4-af20-6684d1875254-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.033260 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.033109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b3cb156-e78f-45e4-af20-6684d1875254-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.033260 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.033143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs22p\" (UniqueName: \"kubernetes.io/projected/1b3cb156-e78f-45e4-af20-6684d1875254-kube-api-access-rs22p\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.033260 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.033189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b3cb156-e78f-45e4-af20-6684d1875254-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.134553 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.134501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b3cb156-e78f-45e4-af20-6684d1875254-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.134700 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.134564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs22p\" (UniqueName: \"kubernetes.io/projected/1b3cb156-e78f-45e4-af20-6684d1875254-kube-api-access-rs22p\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.134700 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.134596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b3cb156-e78f-45e4-af20-6684d1875254-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.134700 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.134653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b3cb156-e78f-45e4-af20-6684d1875254-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.134806 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:34.134762    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 01:26:34.134843 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:34.134825    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3cb156-e78f-45e4-af20-6684d1875254-prometheus-operator-tls podName:1b3cb156-e78f-45e4-af20-6684d1875254 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:34.634808082 +0000 UTC m=+273.305324164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1b3cb156-e78f-45e4-af20-6684d1875254-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-4j6jh" (UID: "1b3cb156-e78f-45e4-af20-6684d1875254") : secret "prometheus-operator-tls" not found
Sep 17 01:26:34.135483 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.135459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b3cb156-e78f-45e4-af20-6684d1875254-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.136984 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.136961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b3cb156-e78f-45e4-af20-6684d1875254-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.142287 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.142269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs22p\" (UniqueName: \"kubernetes.io/projected/1b3cb156-e78f-45e4-af20-6684d1875254-kube-api-access-rs22p\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.638678 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.638632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b3cb156-e78f-45e4-af20-6684d1875254-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.641051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.641028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b3cb156-e78f-45e4-af20-6684d1875254-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4j6jh\" (UID: \"1b3cb156-e78f-45e4-af20-6684d1875254\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:34.887522 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:34.887470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"
Sep 17 01:26:35.152114 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:35.152038    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh"]
Sep 17 01:26:35.154961 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:35.154921    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3cb156_e78f_45e4_af20_6684d1875254.slice/crio-6a06cbbd750803c94dc956775061518cc073ff83e2cb8c49d93dd0d765ee916e WatchSource:0}: Error finding container 6a06cbbd750803c94dc956775061518cc073ff83e2cb8c49d93dd0d765ee916e: Status 404 returned error can't find the container with id 6a06cbbd750803c94dc956775061518cc073ff83e2cb8c49d93dd0d765ee916e
Sep 17 01:26:35.768667 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:35.768630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh" event={"ID":"1b3cb156-e78f-45e4-af20-6684d1875254","Type":"ContainerStarted","Data":"6a06cbbd750803c94dc956775061518cc073ff83e2cb8c49d93dd0d765ee916e"}
Sep 17 01:26:36.773947 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:36.773859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh" event={"ID":"1b3cb156-e78f-45e4-af20-6684d1875254","Type":"ContainerStarted","Data":"54fcdc280d84ed94dbfa7f5733071f23f32fb8f21dedb38e9102852722b218b0"}
Sep 17 01:26:36.773947 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:36.773919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh" event={"ID":"1b3cb156-e78f-45e4-af20-6684d1875254","Type":"ContainerStarted","Data":"10bf07d4ccc22731fdb9535837ca8abfe88ff1f394d20f0637f07b4c40a5c757"}
Sep 17 01:26:36.789972 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:36.789917    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4j6jh" podStartSLOduration=2.457901152 podStartE2EDuration="3.789871531s" totalImagesPullingTime="1.331970379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:36.789390225 +0000 UTC m=+275.459906316" watchObservedRunningTime="2026-09-17 01:26:36.789871531 +0000 UTC m=+275.460387622"
Sep 17 01:26:38.314081 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.314044    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"]
Sep 17 01:26:38.318826 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.318806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.321655 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.321617    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 01:26:38.322757 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.322732    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 01:26:38.322940 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.322732    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 01:26:38.322940 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.322920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-q4dpl\""
Sep 17 01:26:38.330440 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.330412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"]
Sep 17 01:26:38.333682 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.333659    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qq7pq"]
Sep 17 01:26:38.336995 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.336978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.339298 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.339276    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 01:26:38.339408 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.339309    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 01:26:38.339408 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.339309    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 01:26:38.339639 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.339623    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hlk7m\""
Sep 17 01:26:38.366470 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-textfile\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.366589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkkw\" (UniqueName: \"kubernetes.io/projected/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-api-access-8gkkw\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.366589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpzt\" (UniqueName: \"kubernetes.io/projected/bb544b95-5cf3-4468-9fd7-58f785c29c60-kube-api-access-qvpzt\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.366673 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb544b95-5cf3-4468-9fd7-58f785c29c60-root\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.366673 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.366673 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-wtmp\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.366673 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb544b95-5cf3-4468-9fd7-58f785c29c60-sys\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.366806 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.366806 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.366806 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e28578e8-6ade-432c-89e7-fb253a1d8066-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.366907 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.366907 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.366907 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e28578e8-6ade-432c-89e7-fb253a1d8066-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.367009 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-tls\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.367009 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.366967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb544b95-5cf3-4468-9fd7-58f785c29c60-metrics-client-ca\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468174 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb544b95-5cf3-4468-9fd7-58f785c29c60-sys\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468301 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.468301 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468301 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e28578e8-6ade-432c-89e7-fb253a1d8066-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.468301 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.468301 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb544b95-5cf3-4468-9fd7-58f785c29c60-sys\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468569 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.468569 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:38.468382    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 01:26:38.468569 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:38.468443    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-tls podName:e28578e8-6ade-432c-89e7-fb253a1d8066 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:38.968421057 +0000 UTC m=+277.638937139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-bbp4g" (UID: "e28578e8-6ade-432c-89e7-fb253a1d8066") : secret "kube-state-metrics-tls" not found
Sep 17 01:26:38.468734 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e28578e8-6ade-432c-89e7-fb253a1d8066-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.468734 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-tls\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468734 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb544b95-5cf3-4468-9fd7-58f785c29c60-metrics-client-ca\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468734 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e28578e8-6ade-432c-89e7-fb253a1d8066-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.468734 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-textfile\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468988 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkkw\" (UniqueName: \"kubernetes.io/projected/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-api-access-8gkkw\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.468988 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpzt\" (UniqueName: \"kubernetes.io/projected/bb544b95-5cf3-4468-9fd7-58f785c29c60-kube-api-access-qvpzt\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468988 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb544b95-5cf3-4468-9fd7-58f785c29c60-root\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468988 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468988 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-wtmp\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.468988 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.468904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.469640 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.469109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e28578e8-6ade-432c-89e7-fb253a1d8066-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.469640 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.469165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb544b95-5cf3-4468-9fd7-58f785c29c60-root\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.469640 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.469326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb544b95-5cf3-4468-9fd7-58f785c29c60-metrics-client-ca\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.469640 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.469443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-wtmp\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.469640 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.469484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-textfile\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.470120 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.470099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.471443 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.471423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-tls\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.471515 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.471464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.471726 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.471707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb544b95-5cf3-4468-9fd7-58f785c29c60-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.477007 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.476981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpzt\" (UniqueName: \"kubernetes.io/projected/bb544b95-5cf3-4468-9fd7-58f785c29c60-kube-api-access-qvpzt\") pod \"node-exporter-qq7pq\" (UID: \"bb544b95-5cf3-4468-9fd7-58f785c29c60\") " pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.477083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.476982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkkw\" (UniqueName: \"kubernetes.io/projected/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-api-access-8gkkw\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.646552 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.646449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qq7pq"
Sep 17 01:26:38.658455 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:38.658422    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb544b95_5cf3_4468_9fd7_58f785c29c60.slice/crio-6ba4203b22dd206edbcae9b685129b5b5df28c167faf595d0e76303d1c9205a4 WatchSource:0}: Error finding container 6ba4203b22dd206edbcae9b685129b5b5df28c167faf595d0e76303d1c9205a4: Status 404 returned error can't find the container with id 6ba4203b22dd206edbcae9b685129b5b5df28c167faf595d0e76303d1c9205a4
Sep 17 01:26:38.788789 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.788760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq7pq" event={"ID":"bb544b95-5cf3-4468-9fd7-58f785c29c60","Type":"ContainerStarted","Data":"6ba4203b22dd206edbcae9b685129b5b5df28c167faf595d0e76303d1c9205a4"}
Sep 17 01:26:38.973905 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.973818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:38.976134 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:38.976114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e28578e8-6ade-432c-89e7-fb253a1d8066-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-bbp4g\" (UID: \"e28578e8-6ade-432c-89e7-fb253a1d8066\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:39.228147 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:39.228065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"
Sep 17 01:26:39.520919 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:39.520848    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g"]
Sep 17 01:26:39.736606 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:39.736575    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28578e8_6ade_432c_89e7_fb253a1d8066.slice/crio-9493ebf00c6036e62cf515dcd1917f7a3a1d3e686ecc63430e08c4b5f3d804de WatchSource:0}: Error finding container 9493ebf00c6036e62cf515dcd1917f7a3a1d3e686ecc63430e08c4b5f3d804de: Status 404 returned error can't find the container with id 9493ebf00c6036e62cf515dcd1917f7a3a1d3e686ecc63430e08c4b5f3d804de
Sep 17 01:26:39.793806 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:39.793780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g" event={"ID":"e28578e8-6ade-432c-89e7-fb253a1d8066","Type":"ContainerStarted","Data":"9493ebf00c6036e62cf515dcd1917f7a3a1d3e686ecc63430e08c4b5f3d804de"}
Sep 17 01:26:40.799264 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.799229    2578 generic.go:356] "Generic (PLEG): container finished" podID="bb544b95-5cf3-4468-9fd7-58f785c29c60" containerID="e23b9e210c9b0d4496156a652cc2ba3bc7fb8884aaddfe0f0796346a1b2690a9" exitCode=0
Sep 17 01:26:40.799264 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.799267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq7pq" event={"ID":"bb544b95-5cf3-4468-9fd7-58f785c29c60","Type":"ContainerDied","Data":"e23b9e210c9b0d4496156a652cc2ba3bc7fb8884aaddfe0f0796346a1b2690a9"}
Sep 17 01:26:40.856130 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.856099    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69678d4944-2lk2b"]
Sep 17 01:26:40.859365 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.859344    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.862010 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.861990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 01:26:40.862135 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.861995    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 01:26:40.862135 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.862021    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2tnjq\""
Sep 17 01:26:40.862313 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.862297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 01:26:40.862392 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.862360    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 01:26:40.862529 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.862503    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 01:26:40.862651 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.862619    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 01:26:40.862777 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.862673    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 01:26:40.867101 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.866996    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 01:26:40.872027 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.870681    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69678d4944-2lk2b"]
Sep 17 01:26:40.892671 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.892645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95kz\" (UniqueName: \"kubernetes.io/projected/01202739-f65e-454e-a580-193d7719d404-kube-api-access-m95kz\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.892802 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.892680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-oauth-config\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.892802 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.892772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-oauth-serving-cert\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.892909 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.892832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-console-config\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.892909 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.892903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-serving-cert\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.892997 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.892949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-trusted-ca-bundle\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.893056 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.893020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-service-ca\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.993680 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.993639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-trusted-ca-bundle\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.993863 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.993695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-service-ca\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.993927 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.993877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95kz\" (UniqueName: \"kubernetes.io/projected/01202739-f65e-454e-a580-193d7719d404-kube-api-access-m95kz\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.993984 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.993929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-oauth-config\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.994037 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.993987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-oauth-serving-cert\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.994197 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.994173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-console-config\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.994280 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.994231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-serving-cert\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.994419 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.994394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-service-ca\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.994720 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.994675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-oauth-serving-cert\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.994720 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.994687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-trusted-ca-bundle\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.994882 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.994864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-console-config\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.996725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.996676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-oauth-config\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:40.996870 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:40.996849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-serving-cert\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:41.001068 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.001044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95kz\" (UniqueName: \"kubernetes.io/projected/01202739-f65e-454e-a580-193d7719d404-kube-api-access-m95kz\") pod \"console-69678d4944-2lk2b\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") " pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:41.176882 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.176850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:41.466802 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.466774    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69678d4944-2lk2b"]
Sep 17 01:26:41.470122 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:41.470096    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01202739_f65e_454e_a580_193d7719d404.slice/crio-5b308f0be447460402e5d350e1efd0a9b8dda6c54eb01d35adbf429b59394130 WatchSource:0}: Error finding container 5b308f0be447460402e5d350e1efd0a9b8dda6c54eb01d35adbf429b59394130: Status 404 returned error can't find the container with id 5b308f0be447460402e5d350e1efd0a9b8dda6c54eb01d35adbf429b59394130
Sep 17 01:26:41.804733 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.804699    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq7pq" event={"ID":"bb544b95-5cf3-4468-9fd7-58f785c29c60","Type":"ContainerStarted","Data":"ba90365f644c4e5d1b69bf89c3c0ea6baafc9660b6746300af25e821caca4f3b"}
Sep 17 01:26:41.805131 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.804756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq7pq" event={"ID":"bb544b95-5cf3-4468-9fd7-58f785c29c60","Type":"ContainerStarted","Data":"3154742c5e6fd12ee3d5ebddb3d11610364e2bc66d6884490ac7a5e30ab9a23a"}
Sep 17 01:26:41.805962 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.805940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69678d4944-2lk2b" event={"ID":"01202739-f65e-454e-a580-193d7719d404","Type":"ContainerStarted","Data":"5b308f0be447460402e5d350e1efd0a9b8dda6c54eb01d35adbf429b59394130"}
Sep 17 01:26:41.807965 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.807938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g" event={"ID":"e28578e8-6ade-432c-89e7-fb253a1d8066","Type":"ContainerStarted","Data":"6888343fd4516dd53270cf7337216c5a02fa744aa07a0b9d81db3dda5b3d2a8d"}
Sep 17 01:26:41.808061 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.807990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g" event={"ID":"e28578e8-6ade-432c-89e7-fb253a1d8066","Type":"ContainerStarted","Data":"12972033c7ff4e66cc8c4ab20ecc040502438cd749a03ee07805ffec8958d8dd"}
Sep 17 01:26:41.808061 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.808018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g" event={"ID":"e28578e8-6ade-432c-89e7-fb253a1d8066","Type":"ContainerStarted","Data":"3cf4365e3156449a83e6e3a758d7db36b40cb7df143dfa9ccbfacb4633648523"}
Sep 17 01:26:41.822294 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.822253    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qq7pq" podStartSLOduration=2.638755727 podStartE2EDuration="3.822238598s" totalImagesPullingTime="1.121655693s" totalInitContainerRuntime="61.827178ms" isStatefulPod=true podCreationTimestamp="2026-09-17 01:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:41.820064781 +0000 UTC m=+280.490580872" watchObservedRunningTime="2026-09-17 01:26:41.822238598 +0000 UTC m=+280.492754686"
Sep 17 01:26:41.836442 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:41.836397    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-bbp4g" podStartSLOduration=2.38702088 podStartE2EDuration="3.836382865s" totalImagesPullingTime="1.449361985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:41.836005199 +0000 UTC m=+280.506521299" watchObservedRunningTime="2026-09-17 01:26:41.836382865 +0000 UTC m=+280.506898954"
Sep 17 01:26:42.454645 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:42.454608    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nrzkz" podUID="123f13ef-59c4-4635-8550-ee114b9d1b7b"
Sep 17 01:26:42.603761 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.603701    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84754b4c75-25xht"]
Sep 17 01:26:42.606157 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.606132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.608933 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.608905    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 01:26:42.609056 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.609011    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 01:26:42.609237 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.609214    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 01:26:42.609766 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.609702    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-njhpp\""
Sep 17 01:26:42.609920 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.609901    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 01:26:42.610194 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.610168    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1f9f6kfps2im6\""
Sep 17 01:26:42.617054 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.617028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84754b4c75-25xht"]
Sep 17 01:26:42.709521 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.709434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt68l\" (UniqueName: \"kubernetes.io/projected/912ec005-99b4-448a-b189-53ddd78f1893-kube-api-access-zt68l\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.709521 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.709497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/912ec005-99b4-448a-b189-53ddd78f1893-client-ca-bundle\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.709735 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.709577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/912ec005-99b4-448a-b189-53ddd78f1893-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.709735 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.709639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/912ec005-99b4-448a-b189-53ddd78f1893-audit-log\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.709735 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.709702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/912ec005-99b4-448a-b189-53ddd78f1893-secret-metrics-server-tls\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.709735 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.709718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/912ec005-99b4-448a-b189-53ddd78f1893-metrics-server-audit-profiles\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.709890 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.709738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/912ec005-99b4-448a-b189-53ddd78f1893-secret-metrics-server-client-certs\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.810175 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.810139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/912ec005-99b4-448a-b189-53ddd78f1893-secret-metrics-server-tls\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.810175 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.810172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/912ec005-99b4-448a-b189-53ddd78f1893-metrics-server-audit-profiles\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.810713 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.810483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/912ec005-99b4-448a-b189-53ddd78f1893-secret-metrics-server-client-certs\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.810713 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.810559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt68l\" (UniqueName: \"kubernetes.io/projected/912ec005-99b4-448a-b189-53ddd78f1893-kube-api-access-zt68l\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.810713 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.810617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/912ec005-99b4-448a-b189-53ddd78f1893-client-ca-bundle\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.810713 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.810653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/912ec005-99b4-448a-b189-53ddd78f1893-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.810926 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.810728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/912ec005-99b4-448a-b189-53ddd78f1893-audit-log\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.811227 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.811204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/912ec005-99b4-448a-b189-53ddd78f1893-audit-log\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.811380 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.811334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/912ec005-99b4-448a-b189-53ddd78f1893-metrics-server-audit-profiles\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.811575 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.811509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/912ec005-99b4-448a-b189-53ddd78f1893-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.813479 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.813171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:26:42.813479 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.813431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/912ec005-99b4-448a-b189-53ddd78f1893-client-ca-bundle\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.813655 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.813600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/912ec005-99b4-448a-b189-53ddd78f1893-secret-metrics-server-tls\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.813970 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.813949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/912ec005-99b4-448a-b189-53ddd78f1893-secret-metrics-server-client-certs\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.819749 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.819724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt68l\" (UniqueName: \"kubernetes.io/projected/912ec005-99b4-448a-b189-53ddd78f1893-kube-api-access-zt68l\") pod \"metrics-server-84754b4c75-25xht\" (UID: \"912ec005-99b4-448a-b189-53ddd78f1893\") " pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:42.919018 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:42.918981    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:26:43.087399 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.086031    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"]
Sep 17 01:26:43.089443 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.089025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:43.091600 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.091579    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 01:26:43.091990 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.091608    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9kv8n\""
Sep 17 01:26:43.097085 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.097019    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"]
Sep 17 01:26:43.193910 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.193880    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84754b4c75-25xht"]
Sep 17 01:26:43.215291 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.215260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hdh\" (UniqueName: \"kubernetes.io/projected/57bc09b2-56af-49c2-9e8f-52c190a2a917-kube-api-access-85hdh\") pod \"monitoring-plugin-5d85778b9d-rzgsm\" (UID: \"57bc09b2-56af-49c2-9e8f-52c190a2a917\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:43.215468 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.215305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/57bc09b2-56af-49c2-9e8f-52c190a2a917-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rzgsm\" (UID: \"57bc09b2-56af-49c2-9e8f-52c190a2a917\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:43.315999 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.315950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hdh\" (UniqueName: \"kubernetes.io/projected/57bc09b2-56af-49c2-9e8f-52c190a2a917-kube-api-access-85hdh\") pod \"monitoring-plugin-5d85778b9d-rzgsm\" (UID: \"57bc09b2-56af-49c2-9e8f-52c190a2a917\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:43.316179 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.316110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/57bc09b2-56af-49c2-9e8f-52c190a2a917-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rzgsm\" (UID: \"57bc09b2-56af-49c2-9e8f-52c190a2a917\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:43.316296 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:43.316270    2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 01:26:43.316418 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:43.316358    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57bc09b2-56af-49c2-9e8f-52c190a2a917-monitoring-plugin-cert podName:57bc09b2-56af-49c2-9e8f-52c190a2a917 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:43.81633684 +0000 UTC m=+282.486852922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/57bc09b2-56af-49c2-9e8f-52c190a2a917-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-rzgsm" (UID: "57bc09b2-56af-49c2-9e8f-52c190a2a917") : secret "monitoring-plugin-cert" not found
Sep 17 01:26:43.324505 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.324478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hdh\" (UniqueName: \"kubernetes.io/projected/57bc09b2-56af-49c2-9e8f-52c190a2a917-kube-api-access-85hdh\") pod \"monitoring-plugin-5d85778b9d-rzgsm\" (UID: \"57bc09b2-56af-49c2-9e8f-52c190a2a917\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:43.821936 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.821900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/57bc09b2-56af-49c2-9e8f-52c190a2a917-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rzgsm\" (UID: \"57bc09b2-56af-49c2-9e8f-52c190a2a917\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:43.824454 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:43.824430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/57bc09b2-56af-49c2-9e8f-52c190a2a917-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rzgsm\" (UID: \"57bc09b2-56af-49c2-9e8f-52c190a2a917\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:44.002428 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.002393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:44.119783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:44.119700    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912ec005_99b4_448a_b189_53ddd78f1893.slice/crio-f91dd21a33552101ef19d23de387904b3f2846de645528ecf6a80b1be4a14743 WatchSource:0}: Error finding container f91dd21a33552101ef19d23de387904b3f2846de645528ecf6a80b1be4a14743: Status 404 returned error can't find the container with id f91dd21a33552101ef19d23de387904b3f2846de645528ecf6a80b1be4a14743
Sep 17 01:26:44.388522 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.388495    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"]
Sep 17 01:26:44.391229 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:44.391198    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bc09b2_56af_49c2_9e8f_52c190a2a917.slice/crio-9eff5ff43db4a315587f34730e2cc79a9d00b4bf042b0f2bae024ab8cce71e9f WatchSource:0}: Error finding container 9eff5ff43db4a315587f34730e2cc79a9d00b4bf042b0f2bae024ab8cce71e9f: Status 404 returned error can't find the container with id 9eff5ff43db4a315587f34730e2cc79a9d00b4bf042b0f2bae024ab8cce71e9f
Sep 17 01:26:44.515914 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.515882    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 01:26:44.518891 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.518860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.521716 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.521686    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 01:26:44.521855 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.521757    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 01:26:44.521938 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.521895    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 01:26:44.522275 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.522239    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 01:26:44.522395 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.522288    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 01:26:44.522915 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.522808    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 01:26:44.522915 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.522894    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rb4mq\""
Sep 17 01:26:44.523616 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.523336    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 01:26:44.523616 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.523395    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 01:26:44.523616 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.523455    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 01:26:44.523616 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.523455    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 01:26:44.523616 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.523512    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 01:26:44.523897 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.523654    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 01:26:44.523897 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.523673    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7mn4gcpvc3b9\""
Sep 17 01:26:44.523897 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.523877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 01:26:44.525148 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.525127    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 01:26:44.535845 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.535822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 01:26:44.628697 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.628661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.628861 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.628710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.628861 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.628738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.628861 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.628792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bcl5\" (UniqueName: \"kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-kube-api-access-2bcl5\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.628989 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.628856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.628989 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.628903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.628989 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.628960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.628989 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.628982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-web-config\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629151 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-config\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629151 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629151 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629151 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629151 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629151 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-config-out\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.629402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.629290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730409 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730409 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-web-config\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730409 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-config\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730690 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730690 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730690 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730690 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730690 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730690 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730690 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730690 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.730690 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-config-out\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731118 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731118 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731118 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731118 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731118 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731118 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bcl5\" (UniqueName: \"kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-kube-api-access-2bcl5\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731118 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731118 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.730927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731489 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.731178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731489 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.731260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.731742 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.731716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.734941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.733523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-config\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.734941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.733822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.734941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.734353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.734941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.734364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.734941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.734771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.735615 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.735255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.735615 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.735492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.737251 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.736713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.737251 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.737157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.737671 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.737649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.738930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.738848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.738930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.738829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.739078 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.739029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.739157 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.739134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-web-config\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.739297 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.739274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.740174 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.740151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-config-out\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.741475 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.741457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bcl5\" (UniqueName: \"kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-kube-api-access-2bcl5\") pod \"prometheus-k8s-0\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.822966 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.822924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69678d4944-2lk2b" event={"ID":"01202739-f65e-454e-a580-193d7719d404","Type":"ContainerStarted","Data":"9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d"}
Sep 17 01:26:44.824341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.824311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm" event={"ID":"57bc09b2-56af-49c2-9e8f-52c190a2a917","Type":"ContainerStarted","Data":"9eff5ff43db4a315587f34730e2cc79a9d00b4bf042b0f2bae024ab8cce71e9f"}
Sep 17 01:26:44.825728 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.825702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84754b4c75-25xht" event={"ID":"912ec005-99b4-448a-b189-53ddd78f1893","Type":"ContainerStarted","Data":"f91dd21a33552101ef19d23de387904b3f2846de645528ecf6a80b1be4a14743"}
Sep 17 01:26:44.831048 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.830992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:44.839668 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:44.839606    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69678d4944-2lk2b" podStartSLOduration=2.165033719 podStartE2EDuration="4.839583086s" totalImagesPullingTime="2.674549367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:44.83814123 +0000 UTC m=+283.508657319" watchObservedRunningTime="2026-09-17 01:26:44.839583086 +0000 UTC m=+283.510099176"
Sep 17 01:26:45.126247 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:45.126211    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 01:26:45.130195 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:45.130159    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23410cd5_0f31_4783_a59d_1d0371cb882b.slice/crio-35a86f39b923a8c59396a5e443e7070bad25d8503fdd877afb208517e1159fcc WatchSource:0}: Error finding container 35a86f39b923a8c59396a5e443e7070bad25d8503fdd877afb208517e1159fcc: Status 404 returned error can't find the container with id 35a86f39b923a8c59396a5e443e7070bad25d8503fdd877afb208517e1159fcc
Sep 17 01:26:45.642808 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:45.642775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:26:45.645234 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:45.645202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/123f13ef-59c4-4635-8550-ee114b9d1b7b-metrics-tls\") pod \"dns-default-nrzkz\" (UID: \"123f13ef-59c4-4635-8550-ee114b9d1b7b\") " pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:26:45.816478 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:45.816444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sdvq5\""
Sep 17 01:26:45.824486 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:45.824458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:26:45.832084 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:45.832051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerStarted","Data":"35a86f39b923a8c59396a5e443e7070bad25d8503fdd877afb208517e1159fcc"}
Sep 17 01:26:46.262574 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:46.262415    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nrzkz"]
Sep 17 01:26:46.265179 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:26:46.265157    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123f13ef_59c4_4635_8550_ee114b9d1b7b.slice/crio-3a7f85dfcfb2b673a7ebe44d6869e3c2dad5eea4a5e89928440fc76550902d82 WatchSource:0}: Error finding container 3a7f85dfcfb2b673a7ebe44d6869e3c2dad5eea4a5e89928440fc76550902d82: Status 404 returned error can't find the container with id 3a7f85dfcfb2b673a7ebe44d6869e3c2dad5eea4a5e89928440fc76550902d82
Sep 17 01:26:46.837989 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:46.837952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nrzkz" event={"ID":"123f13ef-59c4-4635-8550-ee114b9d1b7b","Type":"ContainerStarted","Data":"3a7f85dfcfb2b673a7ebe44d6869e3c2dad5eea4a5e89928440fc76550902d82"}
Sep 17 01:26:46.839991 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:46.839952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm" event={"ID":"57bc09b2-56af-49c2-9e8f-52c190a2a917","Type":"ContainerStarted","Data":"ffd362cb7ac0e4884b72f0b3d1fea12801bed271ac648c3469115a601542e0d7"}
Sep 17 01:26:46.840160 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:46.840008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:46.842584 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:46.842558    2578 generic.go:356] "Generic (PLEG): container finished" podID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerID="b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d" exitCode=0
Sep 17 01:26:46.842712 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:46.842643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerDied","Data":"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"}
Sep 17 01:26:46.844918 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:46.844893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84754b4c75-25xht" event={"ID":"912ec005-99b4-448a-b189-53ddd78f1893","Type":"ContainerStarted","Data":"13592fb309474abd81e9d3c2b7a1a51c0e356edf8208bfc008f4a6d2cd228dad"}
Sep 17 01:26:46.846326 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:46.846296    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm"
Sep 17 01:26:46.856009 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:46.855964    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rzgsm" podStartSLOduration=2.247497159 podStartE2EDuration="3.855947915s" totalImagesPullingTime="1.608450756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:46.854019143 +0000 UTC m=+285.524535234" watchObservedRunningTime="2026-09-17 01:26:46.855947915 +0000 UTC m=+285.526464007"
Sep 17 01:26:46.884572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:46.884454    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84754b4c75-25xht" podStartSLOduration=3.006087402 podStartE2EDuration="4.884433561s" totalImagesPullingTime="1.878346159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:46.884346997 +0000 UTC m=+285.554863086" watchObservedRunningTime="2026-09-17 01:26:46.884433561 +0000 UTC m=+285.554949654"
Sep 17 01:26:47.067527 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:47.067487    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69678d4944-2lk2b"]
Sep 17 01:26:48.854490 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:48.854448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nrzkz" event={"ID":"123f13ef-59c4-4635-8550-ee114b9d1b7b","Type":"ContainerStarted","Data":"79ea19ef6074ecac299d251011a027dd940c76ac7b8280042a36d166bd9f4928"}
Sep 17 01:26:48.854938 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:48.854530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nrzkz" event={"ID":"123f13ef-59c4-4635-8550-ee114b9d1b7b","Type":"ContainerStarted","Data":"78869840aba40f7176f0da23d324ef23afc569ed05d90ca00395186afd681ba3"}
Sep 17 01:26:48.876142 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:48.875934    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nrzkz" podStartSLOduration=251.866953585 podStartE2EDuration="4m13.875914704s" totalImagesPullingTime="2.008961119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:48.875350534 +0000 UTC m=+287.545866649" watchObservedRunningTime="2026-09-17 01:26:48.875914704 +0000 UTC m=+287.546430796"
Sep 17 01:26:49.859781 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:49.859747    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:26:50.866887 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:50.866851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerStarted","Data":"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"}
Sep 17 01:26:50.867353 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:50.866936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerStarted","Data":"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"}
Sep 17 01:26:51.177028 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:51.176942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:26:52.762284 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:52.762252    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f8798949b-9v5sv"
Sep 17 01:26:52.879997 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:52.879965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerStarted","Data":"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"}
Sep 17 01:26:52.880173 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:52.880028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerStarted","Data":"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"}
Sep 17 01:26:52.880173 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:52.880057    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerStarted","Data":"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"}
Sep 17 01:26:52.880173 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:52.880080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerStarted","Data":"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"}
Sep 17 01:26:52.909514 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:52.909466    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.152821805 podStartE2EDuration="8.909450869s" totalImagesPullingTime="6.745988011s" totalInitContainerRuntime="10.641053ms" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:52.906944042 +0000 UTC m=+291.577460157" watchObservedRunningTime="2026-09-17 01:26:52.909450869 +0000 UTC m=+291.579966962"
Sep 17 01:26:54.832119 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:54.832086    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:26:54.915686 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:26:54.915648    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-blh4d" podUID="ddb0ced7-abc3-439b-b250-ff90f01679cc"
Sep 17 01:26:59.868977 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:26:59.868948    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nrzkz"
Sep 17 01:27:01.756665 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:01.756642    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 01:27:02.919847 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:02.919821    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:27:02.921884 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:02.919867    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:27:05.919601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:05.919574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:27:12.095185 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.095124    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69678d4944-2lk2b" podUID="01202739-f65e-454e-a580-193d7719d404" containerName="console" containerID="cri-o://9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d" gracePeriod=15
Sep 17 01:27:12.334224 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.334196    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:27:12.390426 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390354    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-console-config\") pod \"01202739-f65e-454e-a580-193d7719d404\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") "
Sep 17 01:27:12.390426 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390407    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-serving-cert\") pod \"01202739-f65e-454e-a580-193d7719d404\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") "
Sep 17 01:27:12.390661 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-service-ca\") pod \"01202739-f65e-454e-a580-193d7719d404\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") "
Sep 17 01:27:12.390661 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m95kz\" (UniqueName: \"kubernetes.io/projected/01202739-f65e-454e-a580-193d7719d404-kube-api-access-m95kz\") pod \"01202739-f65e-454e-a580-193d7719d404\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") "
Sep 17 01:27:12.390661 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390572    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-trusted-ca-bundle\") pod \"01202739-f65e-454e-a580-193d7719d404\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") "
Sep 17 01:27:12.390661 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390614    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-oauth-serving-cert\") pod \"01202739-f65e-454e-a580-193d7719d404\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") "
Sep 17 01:27:12.390661 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-oauth-config\") pod \"01202739-f65e-454e-a580-193d7719d404\" (UID: \"01202739-f65e-454e-a580-193d7719d404\") "
Sep 17 01:27:12.390891 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390724    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-console-config" (OuterVolumeSpecName: "console-config") pod "01202739-f65e-454e-a580-193d7719d404" (UID: "01202739-f65e-454e-a580-193d7719d404"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:27:12.390949 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390932    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-console-config\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:27:12.391004 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-service-ca" (OuterVolumeSpecName: "service-ca") pod "01202739-f65e-454e-a580-193d7719d404" (UID: "01202739-f65e-454e-a580-193d7719d404"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:27:12.391004 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.390960    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01202739-f65e-454e-a580-193d7719d404" (UID: "01202739-f65e-454e-a580-193d7719d404"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:27:12.391105 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.391010    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "01202739-f65e-454e-a580-193d7719d404" (UID: "01202739-f65e-454e-a580-193d7719d404"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:27:12.392784 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.392756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01202739-f65e-454e-a580-193d7719d404" (UID: "01202739-f65e-454e-a580-193d7719d404"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:27:12.392877 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.392776    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01202739-f65e-454e-a580-193d7719d404-kube-api-access-m95kz" (OuterVolumeSpecName: "kube-api-access-m95kz") pod "01202739-f65e-454e-a580-193d7719d404" (UID: "01202739-f65e-454e-a580-193d7719d404"). InnerVolumeSpecName "kube-api-access-m95kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:27:12.392877 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.392819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01202739-f65e-454e-a580-193d7719d404" (UID: "01202739-f65e-454e-a580-193d7719d404"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:27:12.491708 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.491678    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-serving-cert\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:27:12.491708 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.491703    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-service-ca\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:27:12.491708 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.491712    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m95kz\" (UniqueName: \"kubernetes.io/projected/01202739-f65e-454e-a580-193d7719d404-kube-api-access-m95kz\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:27:12.491910 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.491721    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-trusted-ca-bundle\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:27:12.491910 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.491729    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01202739-f65e-454e-a580-193d7719d404-oauth-serving-cert\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:27:12.491910 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.491737    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01202739-f65e-454e-a580-193d7719d404-console-oauth-config\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:27:12.952324 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.952293    2578 generic.go:356] "Generic (PLEG): container finished" podID="01202739-f65e-454e-a580-193d7719d404" containerID="9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d" exitCode=0
Sep 17 01:27:12.952500 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.952336    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69678d4944-2lk2b"
Sep 17 01:27:12.952500 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.952380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69678d4944-2lk2b" event={"ID":"01202739-f65e-454e-a580-193d7719d404","Type":"ContainerDied","Data":"9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d"}
Sep 17 01:27:12.952500 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.952429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69678d4944-2lk2b" event={"ID":"01202739-f65e-454e-a580-193d7719d404","Type":"ContainerDied","Data":"5b308f0be447460402e5d350e1efd0a9b8dda6c54eb01d35adbf429b59394130"}
Sep 17 01:27:12.952500 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.952457    2578 scope.go:117] "RemoveContainer" containerID="9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d"
Sep 17 01:27:12.961316 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.961298    2578 scope.go:117] "RemoveContainer" containerID="9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d"
Sep 17 01:27:12.961598 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:27:12.961579    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d\": container with ID starting with 9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d not found: ID does not exist" containerID="9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d"
Sep 17 01:27:12.961664 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.961608    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d"} err="failed to get container status \"9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d\": rpc error: code = NotFound desc = could not find container \"9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d\": container with ID starting with 9da5d72af4ca01f44174577d5b0d18e705ee2a64c8203a1643e7cf10adc0f60d not found: ID does not exist"
Sep 17 01:27:12.971952 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.971930    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69678d4944-2lk2b"]
Sep 17 01:27:12.975272 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:12.975252    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69678d4944-2lk2b"]
Sep 17 01:27:13.919069 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:13.919033    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01202739-f65e-454e-a580-193d7719d404" path="/var/lib/kubelet/pods/01202739-f65e-454e-a580-193d7719d404/volumes"
Sep 17 01:27:22.925015 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:22.924987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:27:22.928931 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:22.928907    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84754b4c75-25xht"
Sep 17 01:27:23.997190 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:23.997161    2578 generic.go:356] "Generic (PLEG): container finished" podID="eae6b120-681a-4868-b818-151648b29629" containerID="180fe54afab6225409c872b58ee5a3ddb9cc4a6b065a8eeee609dde03e8bfb4f" exitCode=0
Sep 17 01:27:23.997592 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:23.997195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qhwv5" event={"ID":"eae6b120-681a-4868-b818-151648b29629","Type":"ContainerDied","Data":"180fe54afab6225409c872b58ee5a3ddb9cc4a6b065a8eeee609dde03e8bfb4f"}
Sep 17 01:27:23.997592 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:23.997501    2578 scope.go:117] "RemoveContainer" containerID="180fe54afab6225409c872b58ee5a3ddb9cc4a6b065a8eeee609dde03e8bfb4f"
Sep 17 01:27:25.003071 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:25.003039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qhwv5" event={"ID":"eae6b120-681a-4868-b818-151648b29629","Type":"ContainerStarted","Data":"0967f4bca8faa203e5707903344f5f4bb16dfde7f1ced85d60e39d9448537ac7"}
Sep 17 01:27:29.018275 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:29.018238    2578 generic.go:356] "Generic (PLEG): container finished" podID="3fb80bee-2f30-457b-9263-6b8a93cf9c4d" containerID="1c66378b2d97f26fc2d3d250b77dee35162d6e50175cc8361a89eee1e017cccc" exitCode=0
Sep 17 01:27:29.019023 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:29.018309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2" event={"ID":"3fb80bee-2f30-457b-9263-6b8a93cf9c4d","Type":"ContainerDied","Data":"1c66378b2d97f26fc2d3d250b77dee35162d6e50175cc8361a89eee1e017cccc"}
Sep 17 01:27:29.019023 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:29.018729    2578 scope.go:117] "RemoveContainer" containerID="1c66378b2d97f26fc2d3d250b77dee35162d6e50175cc8361a89eee1e017cccc"
Sep 17 01:27:30.024875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:30.024848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vv2l2" event={"ID":"3fb80bee-2f30-457b-9263-6b8a93cf9c4d","Type":"ContainerStarted","Data":"483a94e1bc3cd5c2c393339ca499ff6825a12c8e6e6cb0b7e10758de966da677"}
Sep 17 01:27:44.831899 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:44.831864    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:27:44.851178 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:44.851147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:27:45.095130 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:27:45.095045    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:02.849972 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:02.849933    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 01:28:02.850942 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:02.850903    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="prometheus" containerID="cri-o://16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219" gracePeriod=600
Sep 17 01:28:02.851193 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:02.850921    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d" gracePeriod=600
Sep 17 01:28:02.851299 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:02.850947    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy-web" containerID="cri-o://fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7" gracePeriod=600
Sep 17 01:28:02.851373 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:02.850975    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="config-reloader" containerID="cri-o://60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba" gracePeriod=600
Sep 17 01:28:02.851429 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:02.850975    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="thanos-sidecar" containerID="cri-o://1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24" gracePeriod=600
Sep 17 01:28:02.851487 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:02.850976    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy" containerID="cri-o://91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6" gracePeriod=600
Sep 17 01:28:03.095395 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.095369    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.127871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.127767    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-serving-certs-ca-bundle\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.127871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.127826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-thanos-prometheus-http-client-file\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.127871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.127853    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-tls\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.127889    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-metrics-client-ca\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.127920    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-db\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.127970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-metrics-client-certs\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.127994    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-config\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128026    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-tls-assets\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bcl5\" (UniqueName: \"kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-kube-api-access-2bcl5\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128111    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:28:03.128141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128138    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-2\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-config-out\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-grpc-tls\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128270    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-1\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-0\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128320    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-trusted-ca-bundle\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128354    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-web-config\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128382    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-kube-rbac-proxy\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128427    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-kubelet-serving-ca-bundle\") pod \"23410cd5-0f31-4783-a59d-1d0371cb882b\" (UID: \"23410cd5-0f31-4783-a59d-1d0371cb882b\") "
Sep 17 01:28:03.128610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128551    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:28:03.129094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128759    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.129094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128779    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-metrics-client-ca\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.129094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.128876    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:28:03.130869 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.130239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:28:03.130869 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.130784    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:28:03.131044 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.130965    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:28:03.131044 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.131034    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:28:03.131776 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.131482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:28:03.132401 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.132368    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:28:03.132806 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.132782    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:28:03.133990 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.133955    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:28:03.136634 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.136598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:28:03.137475 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.137449    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:28:03.138513 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.138486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:28:03.138625 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.138590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-config" (OuterVolumeSpecName: "config") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:28:03.138787 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.138735    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:28:03.139242 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.139217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-config-out" (OuterVolumeSpecName: "config-out") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:28:03.140082 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.140057    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:28:03.140956 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.140917    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-kube-api-access-2bcl5" (OuterVolumeSpecName: "kube-api-access-2bcl5") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "kube-api-access-2bcl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:28:03.151915 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.151887    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-web-config" (OuterVolumeSpecName: "web-config") pod "23410cd5-0f31-4783-a59d-1d0371cb882b" (UID: "23410cd5-0f31-4783-a59d-1d0371cb882b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:28:03.155155 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155118    2578 generic.go:356] "Generic (PLEG): container finished" podID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerID="88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d" exitCode=0
Sep 17 01:28:03.155155 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155143    2578 generic.go:356] "Generic (PLEG): container finished" podID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerID="91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6" exitCode=0
Sep 17 01:28:03.155155 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155153    2578 generic.go:356] "Generic (PLEG): container finished" podID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerID="fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7" exitCode=0
Sep 17 01:28:03.155383 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155161    2578 generic.go:356] "Generic (PLEG): container finished" podID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerID="1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24" exitCode=0
Sep 17 01:28:03.155383 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155171    2578 generic.go:356] "Generic (PLEG): container finished" podID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerID="60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba" exitCode=0
Sep 17 01:28:03.155383 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155180    2578 generic.go:356] "Generic (PLEG): container finished" podID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerID="16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219" exitCode=0
Sep 17 01:28:03.155383 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerDied","Data":"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"}
Sep 17 01:28:03.155383 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerDied","Data":"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"}
Sep 17 01:28:03.155383 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerDied","Data":"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"}
Sep 17 01:28:03.155383 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerDied","Data":"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"}
Sep 17 01:28:03.155709 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerDied","Data":"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"}
Sep 17 01:28:03.155709 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerDied","Data":"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"}
Sep 17 01:28:03.155709 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23410cd5-0f31-4783-a59d-1d0371cb882b","Type":"ContainerDied","Data":"35a86f39b923a8c59396a5e443e7070bad25d8503fdd877afb208517e1159fcc"}
Sep 17 01:28:03.155709 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155524    2578 scope.go:117] "RemoveContainer" containerID="88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"
Sep 17 01:28:03.155882 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.155745    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.164742 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.164720    2578 scope.go:117] "RemoveContainer" containerID="91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"
Sep 17 01:28:03.172355 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.172337    2578 scope.go:117] "RemoveContainer" containerID="fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"
Sep 17 01:28:03.180063 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.180042    2578 scope.go:117] "RemoveContainer" containerID="1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"
Sep 17 01:28:03.182854 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.182833    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 01:28:03.186779 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.186755    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 01:28:03.188763 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.188747    2578 scope.go:117] "RemoveContainer" containerID="60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"
Sep 17 01:28:03.196608 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.196586    2578 scope.go:117] "RemoveContainer" containerID="16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"
Sep 17 01:28:03.221729 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.203893    2578 scope.go:117] "RemoveContainer" containerID="b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.224831    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225211    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="init-config-reloader"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225226    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="init-config-reloader"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225242    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01202739-f65e-454e-a580-193d7719d404" containerName="console"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225250    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="01202739-f65e-454e-a580-193d7719d404" containerName="console"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225260    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="config-reloader"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225268    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="config-reloader"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225280    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="thanos-sidecar"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225288    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="thanos-sidecar"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225316    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225325    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225334    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy-thanos"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225342    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy-thanos"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225351    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="prometheus"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225360    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="prometheus"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225368    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy-web"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy-web"
Sep 17 01:28:03.225435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225449    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="01202739-f65e-454e-a580-193d7719d404" containerName="console"
Sep 17 01:28:03.226362 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225461    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="config-reloader"
Sep 17 01:28:03.226362 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225469    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy-web"
Sep 17 01:28:03.226362 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225478    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="prometheus"
Sep 17 01:28:03.226362 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.225486    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy-thanos"
Sep 17 01:28:03.226362 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.226331    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="thanos-sidecar"
Sep 17 01:28:03.226362 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.226349    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" containerName="kube-rbac-proxy"
Sep 17 01:28:03.232563 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.232223    2578 scope.go:117] "RemoveContainer" containerID="88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"
Sep 17 01:28:03.232943 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:28:03.232801    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": container with ID starting with 88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d not found: ID does not exist" containerID="88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"
Sep 17 01:28:03.232943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.232837    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"} err="failed to get container status \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": rpc error: code = NotFound desc = could not find container \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": container with ID starting with 88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d not found: ID does not exist"
Sep 17 01:28:03.232943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.232861    2578 scope.go:117] "RemoveContainer" containerID="91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"
Sep 17 01:28:03.232943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.232924    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-db\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.232946    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-metrics-client-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.232961    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-config\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.232976    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.232993    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-tls-assets\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233006    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bcl5\" (UniqueName: \"kubernetes.io/projected/23410cd5-0f31-4783-a59d-1d0371cb882b-kube-api-access-2bcl5\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233018    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233032    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23410cd5-0f31-4783-a59d-1d0371cb882b-config-out\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233045    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-grpc-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233056    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233069    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233082    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233094    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233107    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-web-config\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233120    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-kube-rbac-proxy\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233132    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23410cd5-0f31-4783-a59d-1d0371cb882b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233146    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233160    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23410cd5-0f31-4783-a59d-1d0371cb882b-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:28:03.233941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.233407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.234253 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.234198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 01:28:03.235009 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:28:03.234807    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": container with ID starting with 91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6 not found: ID does not exist" containerID="91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"
Sep 17 01:28:03.235009 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.234860    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"} err="failed to get container status \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": rpc error: code = NotFound desc = could not find container \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": container with ID starting with 91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6 not found: ID does not exist"
Sep 17 01:28:03.235009 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.234884    2578 scope.go:117] "RemoveContainer" containerID="fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"
Sep 17 01:28:03.235293 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:28:03.235271    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": container with ID starting with fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7 not found: ID does not exist" containerID="fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"
Sep 17 01:28:03.235417 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.235299    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"} err="failed to get container status \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": rpc error: code = NotFound desc = could not find container \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": container with ID starting with fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7 not found: ID does not exist"
Sep 17 01:28:03.235417 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.235316    2578 scope.go:117] "RemoveContainer" containerID="1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"
Sep 17 01:28:03.235659 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:28:03.235629    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": container with ID starting with 1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24 not found: ID does not exist" containerID="1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"
Sep 17 01:28:03.235756 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.235658    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"} err="failed to get container status \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": rpc error: code = NotFound desc = could not find container \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": container with ID starting with 1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24 not found: ID does not exist"
Sep 17 01:28:03.235756 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.235679    2578 scope.go:117] "RemoveContainer" containerID="60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"
Sep 17 01:28:03.235987 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:28:03.235970    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": container with ID starting with 60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba not found: ID does not exist" containerID="60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"
Sep 17 01:28:03.236059 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.235991    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"} err="failed to get container status \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": rpc error: code = NotFound desc = could not find container \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": container with ID starting with 60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba not found: ID does not exist"
Sep 17 01:28:03.236059 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.236009    2578 scope.go:117] "RemoveContainer" containerID="16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"
Sep 17 01:28:03.236360 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.236305    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 01:28:03.236360 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:28:03.236303    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": container with ID starting with 16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219 not found: ID does not exist" containerID="16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"
Sep 17 01:28:03.236360 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.236354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 01:28:03.236572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.236351    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"} err="failed to get container status \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": rpc error: code = NotFound desc = could not find container \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": container with ID starting with 16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219 not found: ID does not exist"
Sep 17 01:28:03.236572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.236426    2578 scope.go:117] "RemoveContainer" containerID="b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"
Sep 17 01:28:03.236572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.236561    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 01:28:03.236802 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:28:03.236778    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": container with ID starting with b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d not found: ID does not exist" containerID="b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"
Sep 17 01:28:03.236893 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.236808    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"} err="failed to get container status \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": rpc error: code = NotFound desc = could not find container \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": container with ID starting with b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d not found: ID does not exist"
Sep 17 01:28:03.236893 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.236827    2578 scope.go:117] "RemoveContainer" containerID="88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"
Sep 17 01:28:03.237012 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237001    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 01:28:03.237173 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237148    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 01:28:03.237254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237175    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"} err="failed to get container status \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": rpc error: code = NotFound desc = could not find container \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": container with ID starting with 88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d not found: ID does not exist"
Sep 17 01:28:03.237254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237197    2578 scope.go:117] "RemoveContainer" containerID="91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"
Sep 17 01:28:03.237448 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237430    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 01:28:03.237519 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237481    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"} err="failed to get container status \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": rpc error: code = NotFound desc = could not find container \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": container with ID starting with 91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6 not found: ID does not exist"
Sep 17 01:28:03.237519 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237505    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 01:28:03.237738 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237506    2578 scope.go:117] "RemoveContainer" containerID="fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"
Sep 17 01:28:03.237823 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237808    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 01:28:03.237949 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7mn4gcpvc3b9\""
Sep 17 01:28:03.238039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237944    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"} err="failed to get container status \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": rpc error: code = NotFound desc = could not find container \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": container with ID starting with fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7 not found: ID does not exist"
Sep 17 01:28:03.238039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.237971    2578 scope.go:117] "RemoveContainer" containerID="1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"
Sep 17 01:28:03.238145 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.238095    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 01:28:03.238193 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.238143    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 01:28:03.238237 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.238194    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rb4mq\""
Sep 17 01:28:03.238299 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.238277    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"} err="failed to get container status \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": rpc error: code = NotFound desc = could not find container \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": container with ID starting with 1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24 not found: ID does not exist"
Sep 17 01:28:03.238341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.238302    2578 scope.go:117] "RemoveContainer" containerID="60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"
Sep 17 01:28:03.239117 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.238588    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 01:28:03.239117 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.238601    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 01:28:03.239117 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.238656    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"} err="failed to get container status \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": rpc error: code = NotFound desc = could not find container \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": container with ID starting with 60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba not found: ID does not exist"
Sep 17 01:28:03.239117 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.238679    2578 scope.go:117] "RemoveContainer" containerID="16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"
Sep 17 01:28:03.239378 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.239149    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"} err="failed to get container status \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": rpc error: code = NotFound desc = could not find container \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": container with ID starting with 16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219 not found: ID does not exist"
Sep 17 01:28:03.239378 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.239173    2578 scope.go:117] "RemoveContainer" containerID="b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"
Sep 17 01:28:03.239557 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.239504    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"} err="failed to get container status \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": rpc error: code = NotFound desc = could not find container \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": container with ID starting with b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d not found: ID does not exist"
Sep 17 01:28:03.239622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.239528    2578 scope.go:117] "RemoveContainer" containerID="88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"
Sep 17 01:28:03.239969 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.239835    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"} err="failed to get container status \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": rpc error: code = NotFound desc = could not find container \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": container with ID starting with 88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d not found: ID does not exist"
Sep 17 01:28:03.239969 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.239860    2578 scope.go:117] "RemoveContainer" containerID="91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"
Sep 17 01:28:03.247371 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.240642    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"} err="failed to get container status \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": rpc error: code = NotFound desc = could not find container \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": container with ID starting with 91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6 not found: ID does not exist"
Sep 17 01:28:03.247371 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.240690    2578 scope.go:117] "RemoveContainer" containerID="fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"
Sep 17 01:28:03.247371 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.240992    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 01:28:03.247786 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.247755    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"} err="failed to get container status \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": rpc error: code = NotFound desc = could not find container \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": container with ID starting with fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7 not found: ID does not exist"
Sep 17 01:28:03.247859 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.247789    2578 scope.go:117] "RemoveContainer" containerID="1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"
Sep 17 01:28:03.248055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.248031    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 01:28:03.248728 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.248702    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"} err="failed to get container status \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": rpc error: code = NotFound desc = could not find container \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": container with ID starting with 1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24 not found: ID does not exist"
Sep 17 01:28:03.248728 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.248727    2578 scope.go:117] "RemoveContainer" containerID="60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"
Sep 17 01:28:03.249049 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.249024    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"} err="failed to get container status \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": rpc error: code = NotFound desc = could not find container \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": container with ID starting with 60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba not found: ID does not exist"
Sep 17 01:28:03.249150 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.249051    2578 scope.go:117] "RemoveContainer" containerID="16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"
Sep 17 01:28:03.249465 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.249438    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"} err="failed to get container status \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": rpc error: code = NotFound desc = could not find container \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": container with ID starting with 16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219 not found: ID does not exist"
Sep 17 01:28:03.249605 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.249584    2578 scope.go:117] "RemoveContainer" containerID="b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"
Sep 17 01:28:03.249978 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.249877    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"} err="failed to get container status \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": rpc error: code = NotFound desc = could not find container \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": container with ID starting with b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d not found: ID does not exist"
Sep 17 01:28:03.249978 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.249906    2578 scope.go:117] "RemoveContainer" containerID="88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"
Sep 17 01:28:03.251578 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.251553    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"} err="failed to get container status \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": rpc error: code = NotFound desc = could not find container \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": container with ID starting with 88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d not found: ID does not exist"
Sep 17 01:28:03.251659 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.251579    2578 scope.go:117] "RemoveContainer" containerID="91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"
Sep 17 01:28:03.251892 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.251868    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"} err="failed to get container status \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": rpc error: code = NotFound desc = could not find container \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": container with ID starting with 91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6 not found: ID does not exist"
Sep 17 01:28:03.251945 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.251895    2578 scope.go:117] "RemoveContainer" containerID="fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"
Sep 17 01:28:03.252162 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.252142    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"} err="failed to get container status \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": rpc error: code = NotFound desc = could not find container \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": container with ID starting with fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7 not found: ID does not exist"
Sep 17 01:28:03.252162 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.252160    2578 scope.go:117] "RemoveContainer" containerID="1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"
Sep 17 01:28:03.252389 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.252371    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"} err="failed to get container status \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": rpc error: code = NotFound desc = could not find container \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": container with ID starting with 1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24 not found: ID does not exist"
Sep 17 01:28:03.252427 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.252391    2578 scope.go:117] "RemoveContainer" containerID="60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"
Sep 17 01:28:03.252665 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.252645    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"} err="failed to get container status \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": rpc error: code = NotFound desc = could not find container \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": container with ID starting with 60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba not found: ID does not exist"
Sep 17 01:28:03.252710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.252670    2578 scope.go:117] "RemoveContainer" containerID="16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"
Sep 17 01:28:03.252954 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.252937    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"} err="failed to get container status \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": rpc error: code = NotFound desc = could not find container \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": container with ID starting with 16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219 not found: ID does not exist"
Sep 17 01:28:03.252996 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.252956    2578 scope.go:117] "RemoveContainer" containerID="b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"
Sep 17 01:28:03.253158 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.253143    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"} err="failed to get container status \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": rpc error: code = NotFound desc = could not find container \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": container with ID starting with b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d not found: ID does not exist"
Sep 17 01:28:03.253195 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.253159    2578 scope.go:117] "RemoveContainer" containerID="88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"
Sep 17 01:28:03.253334 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.253318    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"} err="failed to get container status \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": rpc error: code = NotFound desc = could not find container \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": container with ID starting with 88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d not found: ID does not exist"
Sep 17 01:28:03.253382 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.253335    2578 scope.go:117] "RemoveContainer" containerID="91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"
Sep 17 01:28:03.253519 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.253505    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"} err="failed to get container status \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": rpc error: code = NotFound desc = could not find container \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": container with ID starting with 91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6 not found: ID does not exist"
Sep 17 01:28:03.253590 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.253519    2578 scope.go:117] "RemoveContainer" containerID="fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"
Sep 17 01:28:03.253775 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.253756    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"} err="failed to get container status \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": rpc error: code = NotFound desc = could not find container \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": container with ID starting with fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7 not found: ID does not exist"
Sep 17 01:28:03.253854 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.253777    2578 scope.go:117] "RemoveContainer" containerID="1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"
Sep 17 01:28:03.254003 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.253987    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"} err="failed to get container status \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": rpc error: code = NotFound desc = could not find container \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": container with ID starting with 1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24 not found: ID does not exist"
Sep 17 01:28:03.254069 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.254006    2578 scope.go:117] "RemoveContainer" containerID="60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"
Sep 17 01:28:03.254241 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.254219    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"} err="failed to get container status \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": rpc error: code = NotFound desc = could not find container \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": container with ID starting with 60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba not found: ID does not exist"
Sep 17 01:28:03.254285 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.254244    2578 scope.go:117] "RemoveContainer" containerID="16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"
Sep 17 01:28:03.254441 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.254422    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"} err="failed to get container status \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": rpc error: code = NotFound desc = could not find container \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": container with ID starting with 16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219 not found: ID does not exist"
Sep 17 01:28:03.254441 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.254440    2578 scope.go:117] "RemoveContainer" containerID="b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"
Sep 17 01:28:03.254640 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.254625    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"} err="failed to get container status \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": rpc error: code = NotFound desc = could not find container \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": container with ID starting with b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d not found: ID does not exist"
Sep 17 01:28:03.254679 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.254641    2578 scope.go:117] "RemoveContainer" containerID="88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"
Sep 17 01:28:03.254875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.254859    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d"} err="failed to get container status \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": rpc error: code = NotFound desc = could not find container \"88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d\": container with ID starting with 88e07a1dfd58f3b8d136c8edb6253b8a4f31ae48e4289b1fcd8d201175ee5c0d not found: ID does not exist"
Sep 17 01:28:03.254875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.254874    2578 scope.go:117] "RemoveContainer" containerID="91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"
Sep 17 01:28:03.255072 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.255056    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6"} err="failed to get container status \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": rpc error: code = NotFound desc = could not find container \"91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6\": container with ID starting with 91e6576d3793b1f28af82caed367db77f4536983aa2d357390b4e0dbb727fde6 not found: ID does not exist"
Sep 17 01:28:03.255072 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.255071    2578 scope.go:117] "RemoveContainer" containerID="fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"
Sep 17 01:28:03.255247 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.255231    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7"} err="failed to get container status \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": rpc error: code = NotFound desc = could not find container \"fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7\": container with ID starting with fa8b69a38fe769c5ac1af14e8837be31869f3bdf96016c374d246f9da9cc1de7 not found: ID does not exist"
Sep 17 01:28:03.255292 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.255247    2578 scope.go:117] "RemoveContainer" containerID="1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"
Sep 17 01:28:03.255455 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.255439    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24"} err="failed to get container status \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": rpc error: code = NotFound desc = could not find container \"1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24\": container with ID starting with 1e417ff0f591711de91bbc046d9ea3ee87646e133e836c51dd93c3117f81df24 not found: ID does not exist"
Sep 17 01:28:03.255455 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.255454    2578 scope.go:117] "RemoveContainer" containerID="60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"
Sep 17 01:28:03.255651 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.255635    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba"} err="failed to get container status \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": rpc error: code = NotFound desc = could not find container \"60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba\": container with ID starting with 60478a4f1574fd08c4c4d9bd8b9fc5ae0e00afd6b14401236e4d0decbcb60dba not found: ID does not exist"
Sep 17 01:28:03.255651 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.255651    2578 scope.go:117] "RemoveContainer" containerID="16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"
Sep 17 01:28:03.255827 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.255808    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219"} err="failed to get container status \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": rpc error: code = NotFound desc = could not find container \"16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219\": container with ID starting with 16bf1c106036e9e0e986ea39915ed4ed31124788d9f6920d85d82fd0b957c219 not found: ID does not exist"
Sep 17 01:28:03.255878 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.255829    2578 scope.go:117] "RemoveContainer" containerID="b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"
Sep 17 01:28:03.256079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.256060    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d"} err="failed to get container status \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": rpc error: code = NotFound desc = could not find container \"b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d\": container with ID starting with b3f7a27d41f7c8d4bb48628204f948036d50eb9157e882de198ec0cd79cd257d not found: ID does not exist"
Sep 17 01:28:03.334082 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6072a131-09b6-4e4a-9508-b240e512f563-config-out\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334227 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6072a131-09b6-4e4a-9508-b240e512f563-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334227 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334227 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334227 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-web-config\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334227 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334227 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-config\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334227 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334451 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334451 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334451 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334451 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334451 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334451 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334451 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z78zg\" (UniqueName: \"kubernetes.io/projected/6072a131-09b6-4e4a-9508-b240e512f563-kube-api-access-z78zg\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334451 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334791 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334791 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334791 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.334791 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.334617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435493 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6072a131-09b6-4e4a-9508-b240e512f563-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435493 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435493 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435493 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-web-config\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435846 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435846 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-config\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435846 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435846 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435846 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435846 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435846 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435846 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.435846 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.436264 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z78zg\" (UniqueName: \"kubernetes.io/projected/6072a131-09b6-4e4a-9508-b240e512f563-kube-api-access-z78zg\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.436264 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.436264 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.436264 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.436264 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.435993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.436264 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.436028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.436264 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.436067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6072a131-09b6-4e4a-9508-b240e512f563-config-out\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.437215 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.437185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.438440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6072a131-09b6-4e4a-9508-b240e512f563-config-out\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.438627    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-config\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.438668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-web-config\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.438932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6072a131-09b6-4e4a-9508-b240e512f563-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.439192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.439236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.439365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.439489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.439698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.440087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.440203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.440235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440974 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.440311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.440974 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.440496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6072a131-09b6-4e4a-9508-b240e512f563-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.441196 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.441172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.441374 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.441353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.441608 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.441591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.441753 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.441736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6072a131-09b6-4e4a-9508-b240e512f563-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.444765 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.444748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z78zg\" (UniqueName: \"kubernetes.io/projected/6072a131-09b6-4e4a-9508-b240e512f563-kube-api-access-z78zg\") pod \"prometheus-k8s-0\" (UID: \"6072a131-09b6-4e4a-9508-b240e512f563\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.548016 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.547988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:03.823816 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.823791    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 01:28:03.825614 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:28:03.825587    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6072a131_09b6_4e4a_9508_b240e512f563.slice/crio-12991fff4de055cfbc716b2079e5c5c97d4b7039406dc37db6ea17c070597ff4 WatchSource:0}: Error finding container 12991fff4de055cfbc716b2079e5c5c97d4b7039406dc37db6ea17c070597ff4: Status 404 returned error can't find the container with id 12991fff4de055cfbc716b2079e5c5c97d4b7039406dc37db6ea17c070597ff4
Sep 17 01:28:03.920063 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:03.920038    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23410cd5-0f31-4783-a59d-1d0371cb882b" path="/var/lib/kubelet/pods/23410cd5-0f31-4783-a59d-1d0371cb882b/volumes"
Sep 17 01:28:04.161158 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:04.161075    2578 generic.go:356] "Generic (PLEG): container finished" podID="6072a131-09b6-4e4a-9508-b240e512f563" containerID="354277a0cf4449a053e6961a98d7bc31ffa7431355ca48051c5d093f6eb49080" exitCode=0
Sep 17 01:28:04.161158 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:04.161126    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6072a131-09b6-4e4a-9508-b240e512f563","Type":"ContainerDied","Data":"354277a0cf4449a053e6961a98d7bc31ffa7431355ca48051c5d093f6eb49080"}
Sep 17 01:28:04.161329 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:04.161171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6072a131-09b6-4e4a-9508-b240e512f563","Type":"ContainerStarted","Data":"12991fff4de055cfbc716b2079e5c5c97d4b7039406dc37db6ea17c070597ff4"}
Sep 17 01:28:05.169095 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:05.169063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6072a131-09b6-4e4a-9508-b240e512f563","Type":"ContainerStarted","Data":"ae448228f760a0d0ae40a40a7f0317e1942fa801c67486e93f6fc35138edf433"}
Sep 17 01:28:05.169445 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:05.169125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6072a131-09b6-4e4a-9508-b240e512f563","Type":"ContainerStarted","Data":"111eb56bbacc9884167d08a997b5a55735f06d17c7da5cead0e30f6005f84835"}
Sep 17 01:28:05.169445 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:05.169151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6072a131-09b6-4e4a-9508-b240e512f563","Type":"ContainerStarted","Data":"64e6ee07f6da1869b328f7dbc853ae5438eb6da46884b25739033a1863927969"}
Sep 17 01:28:05.169445 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:05.169176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6072a131-09b6-4e4a-9508-b240e512f563","Type":"ContainerStarted","Data":"bbfbb1beac41d0be9cb419857d9a2b8cb88ad26843d1cc79c3d4358e5f4050ca"}
Sep 17 01:28:05.169445 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:05.169204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6072a131-09b6-4e4a-9508-b240e512f563","Type":"ContainerStarted","Data":"3e8715b425d74cb7b40fe7b867f23a6d38150e6ec711c74509705df6c3dcf4ab"}
Sep 17 01:28:05.169445 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:05.169230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6072a131-09b6-4e4a-9508-b240e512f563","Type":"ContainerStarted","Data":"7dfcd18b9a4391304bc642ad1c6b58fa240260a76b307405fcec3a292725f837"}
Sep 17 01:28:08.548638 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:08.548598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:28:15.739413 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:15.739368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:28:15.741771 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:15.741744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb0ced7-abc3-439b-b250-ff90f01679cc-metrics-certs\") pod \"network-metrics-daemon-blh4d\" (UID: \"ddb0ced7-abc3-439b-b250-ff90f01679cc\") " pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:28:15.822315 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:15.822285    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k6hcd\""
Sep 17 01:28:15.830593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:15.830570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blh4d"
Sep 17 01:28:16.095967 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:16.095915    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=13.073840708 podStartE2EDuration="13.095898549s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.057841ms" isStatefulPod=false podCreationTimestamp="2026-09-17 01:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:28:05.195782249 +0000 UTC m=+363.866298352" watchObservedRunningTime="2026-09-17 01:28:16.095898549 +0000 UTC m=+374.766414638"
Sep 17 01:28:16.096962 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:16.096941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-blh4d"]
Sep 17 01:28:16.100575 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:28:16.099384    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb0ced7_abc3_439b_b250_ff90f01679cc.slice/crio-2e758fd9a49284cfb18a63770b41cacfb4f828bbc60058dc13300dfcaf578650 WatchSource:0}: Error finding container 2e758fd9a49284cfb18a63770b41cacfb4f828bbc60058dc13300dfcaf578650: Status 404 returned error can't find the container with id 2e758fd9a49284cfb18a63770b41cacfb4f828bbc60058dc13300dfcaf578650
Sep 17 01:28:16.105895 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:16.105872    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 01:28:16.211512 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:16.211475    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-blh4d" event={"ID":"ddb0ced7-abc3-439b-b250-ff90f01679cc","Type":"ContainerStarted","Data":"2e758fd9a49284cfb18a63770b41cacfb4f828bbc60058dc13300dfcaf578650"}
Sep 17 01:28:17.221051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:17.221013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-blh4d" event={"ID":"ddb0ced7-abc3-439b-b250-ff90f01679cc","Type":"ContainerStarted","Data":"5d00dc2633269dbfe9159dd42c254716468bf799933d6dee25c9b625563c01d8"}
Sep 17 01:28:18.226150 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:18.226119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-blh4d" event={"ID":"ddb0ced7-abc3-439b-b250-ff90f01679cc","Type":"ContainerStarted","Data":"2c0c12f57928efcffcb49f5182f026d703e0ed64cdabb3e6f327f63c055a190a"}
Sep 17 01:28:18.240549 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:28:18.240476    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-blh4d" podStartSLOduration=375.252776566 podStartE2EDuration="6m16.240458746s" totalImagesPullingTime="987.68218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:28:18.239893851 +0000 UTC m=+376.910409932" watchObservedRunningTime="2026-09-17 01:28:18.240458746 +0000 UTC m=+376.910974835"
Sep 17 01:29:03.548973 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:29:03.548944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:29:03.564043 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:29:03.564015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:29:04.416924 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:29:04.416898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 01:30:18.343943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.343910    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"]
Sep 17 01:30:18.347075 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.347059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.349831 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.349805    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 01:30:18.350119 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.350102    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 01:30:18.350933 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.350918    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-29l8d\""
Sep 17 01:30:18.351978 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.351944    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 01:30:18.351978 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.351958    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 01:30:18.352445 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.352425    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 01:30:18.361409 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.361386    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"]
Sep 17 01:30:18.474665 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.474629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/166dd18d-8407-4405-8a6f-7e9085d75e04-cert\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.474848 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.474680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/166dd18d-8407-4405-8a6f-7e9085d75e04-manager-config\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.474848 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.474812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmv6\" (UniqueName: \"kubernetes.io/projected/166dd18d-8407-4405-8a6f-7e9085d75e04-kube-api-access-wvmv6\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.474958 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.474848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/166dd18d-8407-4405-8a6f-7e9085d75e04-metrics-cert\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.575584 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.575549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/166dd18d-8407-4405-8a6f-7e9085d75e04-manager-config\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.575745 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.575632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmv6\" (UniqueName: \"kubernetes.io/projected/166dd18d-8407-4405-8a6f-7e9085d75e04-kube-api-access-wvmv6\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.575745 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.575653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/166dd18d-8407-4405-8a6f-7e9085d75e04-metrics-cert\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.575745 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.575689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/166dd18d-8407-4405-8a6f-7e9085d75e04-cert\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.576192 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.576170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/166dd18d-8407-4405-8a6f-7e9085d75e04-manager-config\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.577918 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.577897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/166dd18d-8407-4405-8a6f-7e9085d75e04-cert\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.577976 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.577963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/166dd18d-8407-4405-8a6f-7e9085d75e04-metrics-cert\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.597923 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.597872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmv6\" (UniqueName: \"kubernetes.io/projected/166dd18d-8407-4405-8a6f-7e9085d75e04-kube-api-access-wvmv6\") pod \"lws-controller-manager-5749dbf97f-plnxb\" (UID: \"166dd18d-8407-4405-8a6f-7e9085d75e04\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.656552 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.656514    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:18.926965 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:18.926942    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"]
Sep 17 01:30:18.930272 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:30:18.930248    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166dd18d_8407_4405_8a6f_7e9085d75e04.slice/crio-a7a0eff2f9e73712807520a266ab074e6ccf56bca0bd5148620c016d6dd1db36 WatchSource:0}: Error finding container a7a0eff2f9e73712807520a266ab074e6ccf56bca0bd5148620c016d6dd1db36: Status 404 returned error can't find the container with id a7a0eff2f9e73712807520a266ab074e6ccf56bca0bd5148620c016d6dd1db36
Sep 17 01:30:19.684527 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:19.684497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb" event={"ID":"166dd18d-8407-4405-8a6f-7e9085d75e04","Type":"ContainerStarted","Data":"a7a0eff2f9e73712807520a266ab074e6ccf56bca0bd5148620c016d6dd1db36"}
Sep 17 01:30:23.702489 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:23.702455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb" event={"ID":"166dd18d-8407-4405-8a6f-7e9085d75e04","Type":"ContainerStarted","Data":"2f4da4b2fa67c204b5c1ae55dda4ee4d31767b6a21f406e7a51ee94799b68d7f"}
Sep 17 01:30:23.702988 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:23.702590    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:23.717640 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:23.717591    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb" podStartSLOduration=1.123647737 podStartE2EDuration="5.717578811s" totalImagesPullingTime="4.593931074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:30:23.716223907 +0000 UTC m=+502.386739993" watchObservedRunningTime="2026-09-17 01:30:23.717578811 +0000 UTC m=+502.388094899"
Sep 17 01:30:31.020482 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.020443    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"]
Sep 17 01:30:31.024019 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.023997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.026556 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.026520    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x4nvm\""
Sep 17 01:30:31.026777 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.026760    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 01:30:31.027261 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.027237    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 01:30:31.027364 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.027239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 01:30:31.027364 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.027280    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 01:30:31.036797 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.036777    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"]
Sep 17 01:30:31.078770 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.078744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.078890 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.078778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-tmp\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.078890 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.078803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-manifests\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.078890 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.078875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.079011 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.078902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grzkj\" (UniqueName: \"kubernetes.io/projected/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-kube-api-access-grzkj\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.179373 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.179348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.179484 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.179379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-tmp\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.179484 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.179398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-manifests\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.179484 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.179417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.179484 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.179435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grzkj\" (UniqueName: \"kubernetes.io/projected/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-kube-api-access-grzkj\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.179828 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.179808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-manifests\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.181374 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.181353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-tmp\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.181809 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.181786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.181887 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.181807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.187350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.187332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grzkj\" (UniqueName: \"kubernetes.io/projected/b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6-kube-api-access-grzkj\") pod \"opendatahub-operator-controller-manager-57c6978585-z7z4j\" (UID: \"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.334334 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.334307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:31.614969 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.610842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"]
Sep 17 01:30:31.616423 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:30:31.616391    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24cfb81_d23a_4bc0_aa2a_fab4d97e73a6.slice/crio-45f4d2a651ccc18872ddb1cf88ad94b52344fa9e08a891f159e133fdf91cf806 WatchSource:0}: Error finding container 45f4d2a651ccc18872ddb1cf88ad94b52344fa9e08a891f159e133fdf91cf806: Status 404 returned error can't find the container with id 45f4d2a651ccc18872ddb1cf88ad94b52344fa9e08a891f159e133fdf91cf806
Sep 17 01:30:31.735674 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:31.735637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j" event={"ID":"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6","Type":"ContainerStarted","Data":"45f4d2a651ccc18872ddb1cf88ad94b52344fa9e08a891f159e133fdf91cf806"}
Sep 17 01:30:34.708835 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:34.708810    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-plnxb"
Sep 17 01:30:34.753245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:34.753215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j" event={"ID":"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6","Type":"ContainerStarted","Data":"0b3ba7053d73a0e6ec6c9a7fcf6656fc746d25711ab779ffb5115190b9f413e1"}
Sep 17 01:30:35.763472 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:35.763432    2578 generic.go:356] "Generic (PLEG): container finished" podID="b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6" containerID="0b3ba7053d73a0e6ec6c9a7fcf6656fc746d25711ab779ffb5115190b9f413e1" exitCode=0
Sep 17 01:30:35.763848 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:35.763517    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j" event={"ID":"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6","Type":"ContainerDied","Data":"0b3ba7053d73a0e6ec6c9a7fcf6656fc746d25711ab779ffb5115190b9f413e1"}
Sep 17 01:30:36.769740 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:36.769697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j" event={"ID":"b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6","Type":"ContainerStarted","Data":"e352327b7bea73e0a7a24cdf8c7ae0a5702f6529e24425f71493b0862d795c82"}
Sep 17 01:30:36.770161 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:36.769828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:36.791386 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:36.791337    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j" podStartSLOduration=2.462495273 podStartE2EDuration="6.791316772s" totalImagesPullingTime="3.326292709s" totalInitContainerRuntime="1.00252879s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:30:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:30:36.789073615 +0000 UTC m=+515.459589779" watchObservedRunningTime="2026-09-17 01:30:36.791316772 +0000 UTC m=+515.461832860"
Sep 17 01:30:47.775617 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:47.775585    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-z7z4j"
Sep 17 01:30:50.560140 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.560106    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"]
Sep 17 01:30:50.680202 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.680154    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"]
Sep 17 01:30:50.680379 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.680287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:50.682721 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.682697    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 01:30:50.682721 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.682717    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8jzlq\""
Sep 17 01:30:50.682937 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.682717    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 01:30:50.682937 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.682827    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 01:30:50.682937 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.682837    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 01:30:50.750946 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.750919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwlt\" (UniqueName: \"kubernetes.io/projected/b6b58fa3-6612-4601-b02c-c35e7e95bad2-kube-api-access-qtwlt\") pod \"kube-auth-proxy-6c5f54bcb9-dtvxf\" (UID: \"b6b58fa3-6612-4601-b02c-c35e7e95bad2\") " pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:50.751104 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.750975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b58fa3-6612-4601-b02c-c35e7e95bad2-tls-certs\") pod \"kube-auth-proxy-6c5f54bcb9-dtvxf\" (UID: \"b6b58fa3-6612-4601-b02c-c35e7e95bad2\") " pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:50.751104 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.751029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6b58fa3-6612-4601-b02c-c35e7e95bad2-tmp\") pod \"kube-auth-proxy-6c5f54bcb9-dtvxf\" (UID: \"b6b58fa3-6612-4601-b02c-c35e7e95bad2\") " pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:50.852132 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.852055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b58fa3-6612-4601-b02c-c35e7e95bad2-tls-certs\") pod \"kube-auth-proxy-6c5f54bcb9-dtvxf\" (UID: \"b6b58fa3-6612-4601-b02c-c35e7e95bad2\") " pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:50.852132 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.852125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6b58fa3-6612-4601-b02c-c35e7e95bad2-tmp\") pod \"kube-auth-proxy-6c5f54bcb9-dtvxf\" (UID: \"b6b58fa3-6612-4601-b02c-c35e7e95bad2\") " pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:50.852364 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.852166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwlt\" (UniqueName: \"kubernetes.io/projected/b6b58fa3-6612-4601-b02c-c35e7e95bad2-kube-api-access-qtwlt\") pod \"kube-auth-proxy-6c5f54bcb9-dtvxf\" (UID: \"b6b58fa3-6612-4601-b02c-c35e7e95bad2\") " pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:50.854362 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.854337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6b58fa3-6612-4601-b02c-c35e7e95bad2-tmp\") pod \"kube-auth-proxy-6c5f54bcb9-dtvxf\" (UID: \"b6b58fa3-6612-4601-b02c-c35e7e95bad2\") " pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:50.854488 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.854472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b58fa3-6612-4601-b02c-c35e7e95bad2-tls-certs\") pod \"kube-auth-proxy-6c5f54bcb9-dtvxf\" (UID: \"b6b58fa3-6612-4601-b02c-c35e7e95bad2\") " pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:50.860377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.860353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwlt\" (UniqueName: \"kubernetes.io/projected/b6b58fa3-6612-4601-b02c-c35e7e95bad2-kube-api-access-qtwlt\") pod \"kube-auth-proxy-6c5f54bcb9-dtvxf\" (UID: \"b6b58fa3-6612-4601-b02c-c35e7e95bad2\") " pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:50.990157 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:50.990127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"
Sep 17 01:30:51.275870 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:51.275845    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf"]
Sep 17 01:30:51.278963 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:30:51.278920    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b58fa3_6612_4601_b02c_c35e7e95bad2.slice/crio-e3f2f88e10c100940f6725a996c0a32a2358faac74c91d519050694808a539c0 WatchSource:0}: Error finding container e3f2f88e10c100940f6725a996c0a32a2358faac74c91d519050694808a539c0: Status 404 returned error can't find the container with id e3f2f88e10c100940f6725a996c0a32a2358faac74c91d519050694808a539c0
Sep 17 01:30:51.831914 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:51.831869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf" event={"ID":"b6b58fa3-6612-4601-b02c-c35e7e95bad2","Type":"ContainerStarted","Data":"e3f2f88e10c100940f6725a996c0a32a2358faac74c91d519050694808a539c0"}
Sep 17 01:30:54.848398 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:54.848361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf" event={"ID":"b6b58fa3-6612-4601-b02c-c35e7e95bad2","Type":"ContainerStarted","Data":"7d62144d85beb8941620494a79f9f4b751bfae0b0624b658264e4ede0512b532"}
Sep 17 01:30:54.863082 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:30:54.863033    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c5f54bcb9-dtvxf" podStartSLOduration=2.164300165 podStartE2EDuration="4.863018481s" totalImagesPullingTime="2.698718316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:30:54.862049698 +0000 UTC m=+533.532565778" watchObservedRunningTime="2026-09-17 01:30:54.863018481 +0000 UTC m=+533.533534569"
Sep 17 01:32:30.504839 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:30.504766    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-8bfsk"]
Sep 17 01:32:30.508135 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:30.508118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-8bfsk"
Sep 17 01:32:30.511457 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:30.511437    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ng2tf\""
Sep 17 01:32:30.511731 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:30.511718    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 01:32:30.511799 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:30.511778    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 01:32:30.522029 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:30.522008    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-8bfsk"]
Sep 17 01:32:30.559682 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:30.559644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8bs5\" (UniqueName: \"kubernetes.io/projected/69cd6c13-d387-474e-80d2-0649d13cfc49-kube-api-access-b8bs5\") pod \"authorino-operator-76b478bd78-8bfsk\" (UID: \"69cd6c13-d387-474e-80d2-0649d13cfc49\") " pod="kuadrant-system/authorino-operator-76b478bd78-8bfsk"
Sep 17 01:32:30.660234 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:30.660194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8bs5\" (UniqueName: \"kubernetes.io/projected/69cd6c13-d387-474e-80d2-0649d13cfc49-kube-api-access-b8bs5\") pod \"authorino-operator-76b478bd78-8bfsk\" (UID: \"69cd6c13-d387-474e-80d2-0649d13cfc49\") " pod="kuadrant-system/authorino-operator-76b478bd78-8bfsk"
Sep 17 01:32:30.670277 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:30.670247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8bs5\" (UniqueName: \"kubernetes.io/projected/69cd6c13-d387-474e-80d2-0649d13cfc49-kube-api-access-b8bs5\") pod \"authorino-operator-76b478bd78-8bfsk\" (UID: \"69cd6c13-d387-474e-80d2-0649d13cfc49\") " pod="kuadrant-system/authorino-operator-76b478bd78-8bfsk"
Sep 17 01:32:30.818109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:30.818077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-8bfsk"
Sep 17 01:32:31.088936 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:31.088912    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-8bfsk"]
Sep 17 01:32:31.091663 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:32:31.091630    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cd6c13_d387_474e_80d2_0649d13cfc49.slice/crio-dd62e7a50f6f5d62b857a4e1624c4f46c2103c6d6e9bd3830f7108e10f3ba903 WatchSource:0}: Error finding container dd62e7a50f6f5d62b857a4e1624c4f46c2103c6d6e9bd3830f7108e10f3ba903: Status 404 returned error can't find the container with id dd62e7a50f6f5d62b857a4e1624c4f46c2103c6d6e9bd3830f7108e10f3ba903
Sep 17 01:32:31.251869 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:31.251830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-8bfsk" event={"ID":"69cd6c13-d387-474e-80d2-0649d13cfc49","Type":"ContainerStarted","Data":"dd62e7a50f6f5d62b857a4e1624c4f46c2103c6d6e9bd3830f7108e10f3ba903"}
Sep 17 01:32:34.268724 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:34.268692    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-8bfsk" event={"ID":"69cd6c13-d387-474e-80d2-0649d13cfc49","Type":"ContainerStarted","Data":"9279c328a0312e10dc3451e09ee29f2f218871057cf9d47da3a8e4413f462b81"}
Sep 17 01:32:34.269220 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:34.268809    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-8bfsk"
Sep 17 01:32:34.285164 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:34.285117    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-8bfsk" podStartSLOduration=1.224021486 podStartE2EDuration="4.285100253s" totalImagesPullingTime="3.061078767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:32:34.284075662 +0000 UTC m=+632.954591763" watchObservedRunningTime="2026-09-17 01:32:34.285100253 +0000 UTC m=+632.955616341"
Sep 17 01:32:45.276795 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:32:45.276760    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-8bfsk"
Sep 17 01:33:25.113550 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.113507    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rd74q"]
Sep 17 01:33:25.117083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.117061    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q"
Sep 17 01:33:25.119387 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.119369    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vq8p7\""
Sep 17 01:33:25.122822 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.122798    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rd74q"]
Sep 17 01:33:25.137365 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.137338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-267cx\" (UniqueName: \"kubernetes.io/projected/a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb-kube-api-access-267cx\") pod \"authorino-7bcfd8f9cd-rd74q\" (UID: \"a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q"
Sep 17 01:33:25.238078 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.238046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-267cx\" (UniqueName: \"kubernetes.io/projected/a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb-kube-api-access-267cx\") pod \"authorino-7bcfd8f9cd-rd74q\" (UID: \"a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q"
Sep 17 01:33:25.240572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.240528    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-6pnkg"]
Sep 17 01:33:25.243642 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.243622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6pnkg"
Sep 17 01:33:25.246995 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.246978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-267cx\" (UniqueName: \"kubernetes.io/projected/a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb-kube-api-access-267cx\") pod \"authorino-7bcfd8f9cd-rd74q\" (UID: \"a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q"
Sep 17 01:33:25.251433 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.251410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6pnkg"]
Sep 17 01:33:25.339180 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.339144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28q9w\" (UniqueName: \"kubernetes.io/projected/3ece203c-b145-43ae-b82e-5e1c55478976-kube-api-access-28q9w\") pod \"authorino-59c974bd48-6pnkg\" (UID: \"3ece203c-b145-43ae-b82e-5e1c55478976\") " pod="kuadrant-system/authorino-59c974bd48-6pnkg"
Sep 17 01:33:25.440633 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.440520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28q9w\" (UniqueName: \"kubernetes.io/projected/3ece203c-b145-43ae-b82e-5e1c55478976-kube-api-access-28q9w\") pod \"authorino-59c974bd48-6pnkg\" (UID: \"3ece203c-b145-43ae-b82e-5e1c55478976\") " pod="kuadrant-system/authorino-59c974bd48-6pnkg"
Sep 17 01:33:25.443378 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.443351    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q"
Sep 17 01:33:25.448145 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.448121    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28q9w\" (UniqueName: \"kubernetes.io/projected/3ece203c-b145-43ae-b82e-5e1c55478976-kube-api-access-28q9w\") pod \"authorino-59c974bd48-6pnkg\" (UID: \"3ece203c-b145-43ae-b82e-5e1c55478976\") " pod="kuadrant-system/authorino-59c974bd48-6pnkg"
Sep 17 01:33:25.561109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.561074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6pnkg"
Sep 17 01:33:25.716980 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.716958    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rd74q"]
Sep 17 01:33:25.718099 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:33:25.718069    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda606ddf9_bc53_4eb2_9d5e_49a57a0a0dcb.slice/crio-375b1871c2476fd9f33370a8ecce3f3c2c88e8c28825670a3d0aa321eb938645 WatchSource:0}: Error finding container 375b1871c2476fd9f33370a8ecce3f3c2c88e8c28825670a3d0aa321eb938645: Status 404 returned error can't find the container with id 375b1871c2476fd9f33370a8ecce3f3c2c88e8c28825670a3d0aa321eb938645
Sep 17 01:33:25.719289 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.719275    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 01:33:25.834295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:25.834272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6pnkg"]
Sep 17 01:33:25.836552 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:33:25.836506    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ece203c_b145_43ae_b82e_5e1c55478976.slice/crio-12bdf242fb7e22a8845de9cb0cc3c9385e64a7d1f0622bba1be32565c7ad6998 WatchSource:0}: Error finding container 12bdf242fb7e22a8845de9cb0cc3c9385e64a7d1f0622bba1be32565c7ad6998: Status 404 returned error can't find the container with id 12bdf242fb7e22a8845de9cb0cc3c9385e64a7d1f0622bba1be32565c7ad6998
Sep 17 01:33:26.492908 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:26.492871    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6pnkg" event={"ID":"3ece203c-b145-43ae-b82e-5e1c55478976","Type":"ContainerStarted","Data":"12bdf242fb7e22a8845de9cb0cc3c9385e64a7d1f0622bba1be32565c7ad6998"}
Sep 17 01:33:26.494038 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:26.494010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q" event={"ID":"a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb","Type":"ContainerStarted","Data":"375b1871c2476fd9f33370a8ecce3f3c2c88e8c28825670a3d0aa321eb938645"}
Sep 17 01:33:28.506188 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:28.506152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6pnkg" event={"ID":"3ece203c-b145-43ae-b82e-5e1c55478976","Type":"ContainerStarted","Data":"1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb"}
Sep 17 01:33:28.507789 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:28.507763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q" event={"ID":"a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb","Type":"ContainerStarted","Data":"350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243"}
Sep 17 01:33:28.518996 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:28.518957    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-6pnkg" podStartSLOduration=1.059202212 podStartE2EDuration="3.518943282s" totalImagesPullingTime="2.45974107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:33:28.518417427 +0000 UTC m=+687.188933521" watchObservedRunningTime="2026-09-17 01:33:28.518943282 +0000 UTC m=+687.189459413"
Sep 17 01:33:28.531049 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:28.530924    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q" podStartSLOduration=0.956853815 podStartE2EDuration="3.530890529s" totalImagesPullingTime="2.574036714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:33:28.530462028 +0000 UTC m=+687.200978136" watchObservedRunningTime="2026-09-17 01:33:28.530890529 +0000 UTC m=+687.201406619"
Sep 17 01:33:28.554374 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:28.554346    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rd74q"]
Sep 17 01:33:30.516042 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:30.515998    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q" podUID="a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb" containerName="authorino" containerID="cri-o://350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243" gracePeriod=30
Sep 17 01:33:30.758149 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:30.758123    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q"
Sep 17 01:33:30.785338 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:30.785249    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-267cx\" (UniqueName: \"kubernetes.io/projected/a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb-kube-api-access-267cx\") pod \"a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb\" (UID: \"a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb\") "
Sep 17 01:33:30.787439 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:30.787411    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb-kube-api-access-267cx" (OuterVolumeSpecName: "kube-api-access-267cx") pod "a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb" (UID: "a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb"). InnerVolumeSpecName "kube-api-access-267cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:33:30.886720 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:30.886682    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-267cx\" (UniqueName: \"kubernetes.io/projected/a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb-kube-api-access-267cx\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:33:31.521632 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:31.521598    2578 generic.go:356] "Generic (PLEG): container finished" podID="a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb" containerID="350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243" exitCode=0
Sep 17 01:33:31.522031 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:31.521638    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q"
Sep 17 01:33:31.522031 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:31.521643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q" event={"ID":"a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb","Type":"ContainerDied","Data":"350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243"}
Sep 17 01:33:31.522031 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:31.521678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rd74q" event={"ID":"a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb","Type":"ContainerDied","Data":"375b1871c2476fd9f33370a8ecce3f3c2c88e8c28825670a3d0aa321eb938645"}
Sep 17 01:33:31.522031 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:31.521699    2578 scope.go:117] "RemoveContainer" containerID="350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243"
Sep 17 01:33:31.532228 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:31.532203    2578 scope.go:117] "RemoveContainer" containerID="350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243"
Sep 17 01:33:31.532521 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:33:31.532501    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243\": container with ID starting with 350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243 not found: ID does not exist" containerID="350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243"
Sep 17 01:33:31.532658 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:31.532526    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243"} err="failed to get container status \"350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243\": rpc error: code = NotFound desc = could not find container \"350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243\": container with ID starting with 350381da406460a835aa4eab805430b1f384a5fd9672461b042b46f9d17b5243 not found: ID does not exist"
Sep 17 01:33:31.544234 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:31.544202    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rd74q"]
Sep 17 01:33:31.546295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:31.546275    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rd74q"]
Sep 17 01:33:31.920556 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:31.920509    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb" path="/var/lib/kubelet/pods/a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb/volumes"
Sep 17 01:33:52.948575 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:52.948528    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-4rr75"]
Sep 17 01:33:52.949141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:52.948872    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb" containerName="authorino"
Sep 17 01:33:52.949141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:52.948884    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb" containerName="authorino"
Sep 17 01:33:52.949141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:52.948940    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a606ddf9-bc53-4eb2-9d5e-49a57a0a0dcb" containerName="authorino"
Sep 17 01:33:52.954979 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:52.954953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4rr75"
Sep 17 01:33:52.957855 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:52.957828    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-4rr75"]
Sep 17 01:33:53.080630 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.080594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6z6x\" (UniqueName: \"kubernetes.io/projected/ef5a2f51-3ebf-4083-9f69-baceb177c404-kube-api-access-c6z6x\") pod \"authorino-557f876f69-4rr75\" (UID: \"ef5a2f51-3ebf-4083-9f69-baceb177c404\") " pod="kuadrant-system/authorino-557f876f69-4rr75"
Sep 17 01:33:53.127581 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.127548    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4rr75"]
Sep 17 01:33:53.127818 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:33:53.127799    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-c6z6x], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-4rr75" podUID="ef5a2f51-3ebf-4083-9f69-baceb177c404"
Sep 17 01:33:53.150388 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.150358    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t9rh9"]
Sep 17 01:33:53.154565 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.154524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t9rh9"
Sep 17 01:33:53.157561 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.157486    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 01:33:53.159873 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.159852    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t9rh9"]
Sep 17 01:33:53.181698 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.181661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6z6x\" (UniqueName: \"kubernetes.io/projected/ef5a2f51-3ebf-4083-9f69-baceb177c404-kube-api-access-c6z6x\") pod \"authorino-557f876f69-4rr75\" (UID: \"ef5a2f51-3ebf-4083-9f69-baceb177c404\") " pod="kuadrant-system/authorino-557f876f69-4rr75"
Sep 17 01:33:53.193727 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.193707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6z6x\" (UniqueName: \"kubernetes.io/projected/ef5a2f51-3ebf-4083-9f69-baceb177c404-kube-api-access-c6z6x\") pod \"authorino-557f876f69-4rr75\" (UID: \"ef5a2f51-3ebf-4083-9f69-baceb177c404\") " pod="kuadrant-system/authorino-557f876f69-4rr75"
Sep 17 01:33:53.206591 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.206529    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t9rh9"]
Sep 17 01:33:53.206795 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:33:53.206778    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-m7np4 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-m7np4 tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-t9rh9" podUID="7fef41e8-eeaf-4157-87c7-65de273525b3"
Sep 17 01:33:53.227166 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.227140    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fc799b9bd-4s2nf"]
Sep 17 01:33:53.231129 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.231112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc799b9bd-4s2nf"
Sep 17 01:33:53.234613 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.234591    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc799b9bd-4s2nf"]
Sep 17 01:33:53.282893 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.282861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7np4\" (UniqueName: \"kubernetes.io/projected/7fef41e8-eeaf-4157-87c7-65de273525b3-kube-api-access-m7np4\") pod \"authorino-6d4dfc649f-t9rh9\" (UID: \"7fef41e8-eeaf-4157-87c7-65de273525b3\") " pod="kuadrant-system/authorino-6d4dfc649f-t9rh9"
Sep 17 01:33:53.283053 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.282904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7fef41e8-eeaf-4157-87c7-65de273525b3-tls-cert\") pod \"authorino-6d4dfc649f-t9rh9\" (UID: \"7fef41e8-eeaf-4157-87c7-65de273525b3\") " pod="kuadrant-system/authorino-6d4dfc649f-t9rh9"
Sep 17 01:33:53.383380 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.383340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7np4\" (UniqueName: \"kubernetes.io/projected/7fef41e8-eeaf-4157-87c7-65de273525b3-kube-api-access-m7np4\") pod \"authorino-6d4dfc649f-t9rh9\" (UID: \"7fef41e8-eeaf-4157-87c7-65de273525b3\") " pod="kuadrant-system/authorino-6d4dfc649f-t9rh9"
Sep 17 01:33:53.383596 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.383391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c6b3327-a477-466e-8b07-6d4a3deda193-tls-cert\") pod \"authorino-5fc799b9bd-4s2nf\" (UID: \"4c6b3327-a477-466e-8b07-6d4a3deda193\") " pod="kuadrant-system/authorino-5fc799b9bd-4s2nf"
Sep 17 01:33:53.383596 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.383427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7fef41e8-eeaf-4157-87c7-65de273525b3-tls-cert\") pod \"authorino-6d4dfc649f-t9rh9\" (UID: \"7fef41e8-eeaf-4157-87c7-65de273525b3\") " pod="kuadrant-system/authorino-6d4dfc649f-t9rh9"
Sep 17 01:33:53.383596 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.383506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qlf\" (UniqueName: \"kubernetes.io/projected/4c6b3327-a477-466e-8b07-6d4a3deda193-kube-api-access-45qlf\") pod \"authorino-5fc799b9bd-4s2nf\" (UID: \"4c6b3327-a477-466e-8b07-6d4a3deda193\") " pod="kuadrant-system/authorino-5fc799b9bd-4s2nf"
Sep 17 01:33:53.385845 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.385822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7fef41e8-eeaf-4157-87c7-65de273525b3-tls-cert\") pod \"authorino-6d4dfc649f-t9rh9\" (UID: \"7fef41e8-eeaf-4157-87c7-65de273525b3\") " pod="kuadrant-system/authorino-6d4dfc649f-t9rh9"
Sep 17 01:33:53.390568 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.390530    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7np4\" (UniqueName: \"kubernetes.io/projected/7fef41e8-eeaf-4157-87c7-65de273525b3-kube-api-access-m7np4\") pod \"authorino-6d4dfc649f-t9rh9\" (UID: \"7fef41e8-eeaf-4157-87c7-65de273525b3\") " pod="kuadrant-system/authorino-6d4dfc649f-t9rh9"
Sep 17 01:33:53.484344 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.484258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qlf\" (UniqueName: \"kubernetes.io/projected/4c6b3327-a477-466e-8b07-6d4a3deda193-kube-api-access-45qlf\") pod \"authorino-5fc799b9bd-4s2nf\" (UID: \"4c6b3327-a477-466e-8b07-6d4a3deda193\") " pod="kuadrant-system/authorino-5fc799b9bd-4s2nf"
Sep 17 01:33:53.484497 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.484359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c6b3327-a477-466e-8b07-6d4a3deda193-tls-cert\") pod \"authorino-5fc799b9bd-4s2nf\" (UID: \"4c6b3327-a477-466e-8b07-6d4a3deda193\") " pod="kuadrant-system/authorino-5fc799b9bd-4s2nf"
Sep 17 01:33:53.486704 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.486680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c6b3327-a477-466e-8b07-6d4a3deda193-tls-cert\") pod \"authorino-5fc799b9bd-4s2nf\" (UID: \"4c6b3327-a477-466e-8b07-6d4a3deda193\") " pod="kuadrant-system/authorino-5fc799b9bd-4s2nf"
Sep 17 01:33:53.491434 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.491414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qlf\" (UniqueName: \"kubernetes.io/projected/4c6b3327-a477-466e-8b07-6d4a3deda193-kube-api-access-45qlf\") pod \"authorino-5fc799b9bd-4s2nf\" (UID: \"4c6b3327-a477-466e-8b07-6d4a3deda193\") " pod="kuadrant-system/authorino-5fc799b9bd-4s2nf"
Sep 17 01:33:53.542147 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.542121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc799b9bd-4s2nf"
Sep 17 01:33:53.623723 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.623697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4rr75"
Sep 17 01:33:53.623852 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.623697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t9rh9"
Sep 17 01:33:53.633008 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.632987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4rr75"
Sep 17 01:33:53.635718 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.635684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t9rh9"
Sep 17 01:33:53.787487 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.787386    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6z6x\" (UniqueName: \"kubernetes.io/projected/ef5a2f51-3ebf-4083-9f69-baceb177c404-kube-api-access-c6z6x\") pod \"ef5a2f51-3ebf-4083-9f69-baceb177c404\" (UID: \"ef5a2f51-3ebf-4083-9f69-baceb177c404\") "
Sep 17 01:33:53.787686 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.787499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7np4\" (UniqueName: \"kubernetes.io/projected/7fef41e8-eeaf-4157-87c7-65de273525b3-kube-api-access-m7np4\") pod \"7fef41e8-eeaf-4157-87c7-65de273525b3\" (UID: \"7fef41e8-eeaf-4157-87c7-65de273525b3\") "
Sep 17 01:33:53.787686 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.787557    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7fef41e8-eeaf-4157-87c7-65de273525b3-tls-cert\") pod \"7fef41e8-eeaf-4157-87c7-65de273525b3\" (UID: \"7fef41e8-eeaf-4157-87c7-65de273525b3\") "
Sep 17 01:33:53.789625 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.789592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fef41e8-eeaf-4157-87c7-65de273525b3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7fef41e8-eeaf-4157-87c7-65de273525b3" (UID: "7fef41e8-eeaf-4157-87c7-65de273525b3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:33:53.789706 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.789618    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5a2f51-3ebf-4083-9f69-baceb177c404-kube-api-access-c6z6x" (OuterVolumeSpecName: "kube-api-access-c6z6x") pod "ef5a2f51-3ebf-4083-9f69-baceb177c404" (UID: "ef5a2f51-3ebf-4083-9f69-baceb177c404"). InnerVolumeSpecName "kube-api-access-c6z6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:33:53.789706 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.789673    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fef41e8-eeaf-4157-87c7-65de273525b3-kube-api-access-m7np4" (OuterVolumeSpecName: "kube-api-access-m7np4") pod "7fef41e8-eeaf-4157-87c7-65de273525b3" (UID: "7fef41e8-eeaf-4157-87c7-65de273525b3"). InnerVolumeSpecName "kube-api-access-m7np4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:33:53.808249 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.808227    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc799b9bd-4s2nf"]
Sep 17 01:33:53.810588 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:33:53.810565    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c6b3327_a477_466e_8b07_6d4a3deda193.slice/crio-037d53597c361e2630219dbddb0e8fd36c0254924d28a4aaa86dd4004aa2d20b WatchSource:0}: Error finding container 037d53597c361e2630219dbddb0e8fd36c0254924d28a4aaa86dd4004aa2d20b: Status 404 returned error can't find the container with id 037d53597c361e2630219dbddb0e8fd36c0254924d28a4aaa86dd4004aa2d20b
Sep 17 01:33:53.888988 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.888960    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7np4\" (UniqueName: \"kubernetes.io/projected/7fef41e8-eeaf-4157-87c7-65de273525b3-kube-api-access-m7np4\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:33:53.888988 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.888983    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7fef41e8-eeaf-4157-87c7-65de273525b3-tls-cert\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:33:53.888988 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:53.888994    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6z6x\" (UniqueName: \"kubernetes.io/projected/ef5a2f51-3ebf-4083-9f69-baceb177c404-kube-api-access-c6z6x\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:33:54.629329 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:54.629291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t9rh9"
Sep 17 01:33:54.629329 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:54.629319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4rr75"
Sep 17 01:33:54.629848 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:54.629293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc799b9bd-4s2nf" event={"ID":"4c6b3327-a477-466e-8b07-6d4a3deda193","Type":"ContainerStarted","Data":"037d53597c361e2630219dbddb0e8fd36c0254924d28a4aaa86dd4004aa2d20b"}
Sep 17 01:33:54.652241 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:54.652213    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t9rh9"]
Sep 17 01:33:54.655154 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:54.655126    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t9rh9"]
Sep 17 01:33:54.671365 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:54.671327    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4rr75"]
Sep 17 01:33:54.673012 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:54.672991    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-4rr75"]
Sep 17 01:33:55.635292 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:55.635257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc799b9bd-4s2nf" event={"ID":"4c6b3327-a477-466e-8b07-6d4a3deda193","Type":"ContainerStarted","Data":"49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99"}
Sep 17 01:33:55.650769 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:55.650721    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5fc799b9bd-4s2nf" podStartSLOduration=1.540212191 podStartE2EDuration="2.650705812s" totalImagesPullingTime="1.110493621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:33:55.649686636 +0000 UTC m=+714.320202726" watchObservedRunningTime="2026-09-17 01:33:55.650705812 +0000 UTC m=+714.321221933"
Sep 17 01:33:55.673985 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:55.673957    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6pnkg"]
Sep 17 01:33:55.674180 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:55.674157    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-6pnkg" podUID="3ece203c-b145-43ae-b82e-5e1c55478976" containerName="authorino" containerID="cri-o://1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb" gracePeriod=30
Sep 17 01:33:55.919964 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:55.919937    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fef41e8-eeaf-4157-87c7-65de273525b3" path="/var/lib/kubelet/pods/7fef41e8-eeaf-4157-87c7-65de273525b3/volumes"
Sep 17 01:33:55.920168 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:55.920156    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef5a2f51-3ebf-4083-9f69-baceb177c404" path="/var/lib/kubelet/pods/ef5a2f51-3ebf-4083-9f69-baceb177c404/volumes"
Sep 17 01:33:55.928529 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:55.928510    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6pnkg"
Sep 17 01:33:56.004724 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.004693    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28q9w\" (UniqueName: \"kubernetes.io/projected/3ece203c-b145-43ae-b82e-5e1c55478976-kube-api-access-28q9w\") pod \"3ece203c-b145-43ae-b82e-5e1c55478976\" (UID: \"3ece203c-b145-43ae-b82e-5e1c55478976\") "
Sep 17 01:33:56.006709 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.006684    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ece203c-b145-43ae-b82e-5e1c55478976-kube-api-access-28q9w" (OuterVolumeSpecName: "kube-api-access-28q9w") pod "3ece203c-b145-43ae-b82e-5e1c55478976" (UID: "3ece203c-b145-43ae-b82e-5e1c55478976"). InnerVolumeSpecName "kube-api-access-28q9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:33:56.105488 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.105453    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28q9w\" (UniqueName: \"kubernetes.io/projected/3ece203c-b145-43ae-b82e-5e1c55478976-kube-api-access-28q9w\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:33:56.641206 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.641176    2578 generic.go:356] "Generic (PLEG): container finished" podID="3ece203c-b145-43ae-b82e-5e1c55478976" containerID="1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb" exitCode=0
Sep 17 01:33:56.641601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.641213    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6pnkg"
Sep 17 01:33:56.641601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.641264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6pnkg" event={"ID":"3ece203c-b145-43ae-b82e-5e1c55478976","Type":"ContainerDied","Data":"1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb"}
Sep 17 01:33:56.641601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.641305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6pnkg" event={"ID":"3ece203c-b145-43ae-b82e-5e1c55478976","Type":"ContainerDied","Data":"12bdf242fb7e22a8845de9cb0cc3c9385e64a7d1f0622bba1be32565c7ad6998"}
Sep 17 01:33:56.641601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.641326    2578 scope.go:117] "RemoveContainer" containerID="1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb"
Sep 17 01:33:56.649625 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.649608    2578 scope.go:117] "RemoveContainer" containerID="1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb"
Sep 17 01:33:56.649890 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:33:56.649873    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb\": container with ID starting with 1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb not found: ID does not exist" containerID="1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb"
Sep 17 01:33:56.649936 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.649900    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb"} err="failed to get container status \"1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb\": rpc error: code = NotFound desc = could not find container \"1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb\": container with ID starting with 1190292b16eb74236c8fd9d8d5e8c5a188ab013d6385de5852dbae5931f9fbfb not found: ID does not exist"
Sep 17 01:33:56.659903 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.659880    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6pnkg"]
Sep 17 01:33:56.663486 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:56.663461    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6pnkg"]
Sep 17 01:33:57.920010 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:33:57.919977    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ece203c-b145-43ae-b82e-5e1c55478976" path="/var/lib/kubelet/pods/3ece203c-b145-43ae-b82e-5e1c55478976/volumes"
Sep 17 01:34:00.366944 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.366910    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-m26ps"]
Sep 17 01:34:00.367390 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.367357    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ece203c-b145-43ae-b82e-5e1c55478976" containerName="authorino"
Sep 17 01:34:00.367390 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.367373    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ece203c-b145-43ae-b82e-5e1c55478976" containerName="authorino"
Sep 17 01:34:00.367523 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.367508    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ece203c-b145-43ae-b82e-5e1c55478976" containerName="authorino"
Sep 17 01:34:00.370585 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.370561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.373140 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.373118    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 01:34:00.373308 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.373286    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 01:34:00.373425 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.373238    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 01:34:00.373481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.373210    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fwpg7\""
Sep 17 01:34:00.381909 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.381881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-m26ps"]
Sep 17 01:34:00.546742 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.546706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-webhook-cert\") pod \"maas-controller-75dc6cbdf-m26ps\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") " pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.546742 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.546745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpsv\" (UniqueName: \"kubernetes.io/projected/e1e6396b-c12b-42f0-a708-ec2b7d380019-kube-api-access-8jpsv\") pod \"maas-controller-75dc6cbdf-m26ps\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") " pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.546935 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.546831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-metrics-certs\") pod \"maas-controller-75dc6cbdf-m26ps\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") " pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.647345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.647254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-webhook-cert\") pod \"maas-controller-75dc6cbdf-m26ps\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") " pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.647345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.647295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpsv\" (UniqueName: \"kubernetes.io/projected/e1e6396b-c12b-42f0-a708-ec2b7d380019-kube-api-access-8jpsv\") pod \"maas-controller-75dc6cbdf-m26ps\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") " pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.647587 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.647355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-metrics-certs\") pod \"maas-controller-75dc6cbdf-m26ps\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") " pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.650063 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.650038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-webhook-cert\") pod \"maas-controller-75dc6cbdf-m26ps\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") " pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.650161 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.650038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-metrics-certs\") pod \"maas-controller-75dc6cbdf-m26ps\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") " pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.654591 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.654574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpsv\" (UniqueName: \"kubernetes.io/projected/e1e6396b-c12b-42f0-a708-ec2b7d380019-kube-api-access-8jpsv\") pod \"maas-controller-75dc6cbdf-m26ps\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") " pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.683135 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.683104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:00.950399 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:00.950314    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-m26ps"]
Sep 17 01:34:00.953145 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:34:00.953115    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6396b_c12b_42f0_a708_ec2b7d380019.slice/crio-f0331663981b31fab2ec347ec88ac08d8ba2457ea4ac965fb9d9fa3a830ff41f WatchSource:0}: Error finding container f0331663981b31fab2ec347ec88ac08d8ba2457ea4ac965fb9d9fa3a830ff41f: Status 404 returned error can't find the container with id f0331663981b31fab2ec347ec88ac08d8ba2457ea4ac965fb9d9fa3a830ff41f
Sep 17 01:34:01.672246 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:01.672202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-m26ps" event={"ID":"e1e6396b-c12b-42f0-a708-ec2b7d380019","Type":"ContainerStarted","Data":"f0331663981b31fab2ec347ec88ac08d8ba2457ea4ac965fb9d9fa3a830ff41f"}
Sep 17 01:34:03.194780 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:03.194756    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 01:34:03.683876 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:03.683843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-m26ps" event={"ID":"e1e6396b-c12b-42f0-a708-ec2b7d380019","Type":"ContainerStarted","Data":"ad48807f2ee6303b76240b941d8a692b87612c59f9dd9c4080763eb693632183"}
Sep 17 01:34:03.684082 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:03.684041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:03.701951 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:03.701901    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75dc6cbdf-m26ps" podStartSLOduration=1.463825909 podStartE2EDuration="3.701884614s" totalImagesPullingTime="2.238058705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:34:03.700104026 +0000 UTC m=+722.370620138" watchObservedRunningTime="2026-09-17 01:34:03.701884614 +0000 UTC m=+722.372400702"
Sep 17 01:34:14.517999 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.517967    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-m26ps"]
Sep 17 01:34:14.518347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.518156    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75dc6cbdf-m26ps" podUID="e1e6396b-c12b-42f0-a708-ec2b7d380019" containerName="manager" containerID="cri-o://ad48807f2ee6303b76240b941d8a692b87612c59f9dd9c4080763eb693632183" gracePeriod=30
Sep 17 01:34:14.519735 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.519708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:14.741088 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.741061    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1e6396b-c12b-42f0-a708-ec2b7d380019" containerID="ad48807f2ee6303b76240b941d8a692b87612c59f9dd9c4080763eb693632183" exitCode=0
Sep 17 01:34:14.741246 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.741141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-m26ps" event={"ID":"e1e6396b-c12b-42f0-a708-ec2b7d380019","Type":"ContainerDied","Data":"ad48807f2ee6303b76240b941d8a692b87612c59f9dd9c4080763eb693632183"}
Sep 17 01:34:14.760757 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.760735    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:14.874855 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.874819    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-webhook-cert\") pod \"e1e6396b-c12b-42f0-a708-ec2b7d380019\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") "
Sep 17 01:34:14.875038 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.874872    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jpsv\" (UniqueName: \"kubernetes.io/projected/e1e6396b-c12b-42f0-a708-ec2b7d380019-kube-api-access-8jpsv\") pod \"e1e6396b-c12b-42f0-a708-ec2b7d380019\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") "
Sep 17 01:34:14.875038 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.874903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-metrics-certs\") pod \"e1e6396b-c12b-42f0-a708-ec2b7d380019\" (UID: \"e1e6396b-c12b-42f0-a708-ec2b7d380019\") "
Sep 17 01:34:14.877030 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.877002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e1e6396b-c12b-42f0-a708-ec2b7d380019" (UID: "e1e6396b-c12b-42f0-a708-ec2b7d380019"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:34:14.877147 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.877125    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e1e6396b-c12b-42f0-a708-ec2b7d380019" (UID: "e1e6396b-c12b-42f0-a708-ec2b7d380019"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:34:14.877147 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.877137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e6396b-c12b-42f0-a708-ec2b7d380019-kube-api-access-8jpsv" (OuterVolumeSpecName: "kube-api-access-8jpsv") pod "e1e6396b-c12b-42f0-a708-ec2b7d380019" (UID: "e1e6396b-c12b-42f0-a708-ec2b7d380019"). InnerVolumeSpecName "kube-api-access-8jpsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:34:14.975710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.975672    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-webhook-cert\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:34:14.975710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.975702    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jpsv\" (UniqueName: \"kubernetes.io/projected/e1e6396b-c12b-42f0-a708-ec2b7d380019-kube-api-access-8jpsv\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:34:14.975710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:14.975713    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e6396b-c12b-42f0-a708-ec2b7d380019-metrics-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:34:15.746883 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:15.746842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-m26ps" event={"ID":"e1e6396b-c12b-42f0-a708-ec2b7d380019","Type":"ContainerDied","Data":"f0331663981b31fab2ec347ec88ac08d8ba2457ea4ac965fb9d9fa3a830ff41f"}
Sep 17 01:34:15.747279 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:15.746909    2578 scope.go:117] "RemoveContainer" containerID="ad48807f2ee6303b76240b941d8a692b87612c59f9dd9c4080763eb693632183"
Sep 17 01:34:15.747279 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:15.746857    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-m26ps"
Sep 17 01:34:15.766165 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:15.766137    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-m26ps"]
Sep 17 01:34:15.768031 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:15.768012    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-m26ps"]
Sep 17 01:34:15.919289 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:15.919260    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e6396b-c12b-42f0-a708-ec2b7d380019" path="/var/lib/kubelet/pods/e1e6396b-c12b-42f0-a708-ec2b7d380019/volumes"
Sep 17 01:34:36.488121 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.488089    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"]
Sep 17 01:34:36.488593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.488415    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e6396b-c12b-42f0-a708-ec2b7d380019" containerName="manager"
Sep 17 01:34:36.488593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.488426    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e6396b-c12b-42f0-a708-ec2b7d380019" containerName="manager"
Sep 17 01:34:36.488593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.488481    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e6396b-c12b-42f0-a708-ec2b7d380019" containerName="manager"
Sep 17 01:34:36.494837 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.494819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.498296 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.498263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 01:34:36.498296 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.498283    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 01:34:36.498462 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.498364    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 01:34:36.498462 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.498385    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s7lw7\""
Sep 17 01:34:36.498462 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.498411    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 17 01:34:36.503045 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.502919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"]
Sep 17 01:34:36.569932 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.569894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls5gt\" (UniqueName: \"kubernetes.io/projected/0590fb3d-04f3-4064-a911-3fc95c92d7d1-kube-api-access-ls5gt\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.570100 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.569951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0590fb3d-04f3-4064-a911-3fc95c92d7d1-openshift-service-ca\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.570100 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.570035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-metrics-certs\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.570100 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.570077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-maas-api-tls\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.671162 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.671125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0590fb3d-04f3-4064-a911-3fc95c92d7d1-openshift-service-ca\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.671162 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.671168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-metrics-certs\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.671377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.671189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-maas-api-tls\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.671377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.671243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls5gt\" (UniqueName: \"kubernetes.io/projected/0590fb3d-04f3-4064-a911-3fc95c92d7d1-kube-api-access-ls5gt\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.671377 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:34:36.671300    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 17 01:34:36.671493 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:34:36.671398    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-metrics-certs podName:0590fb3d-04f3-4064-a911-3fc95c92d7d1 nodeName:}" failed. No retries permitted until 2026-09-17 01:34:37.171380844 +0000 UTC m=+755.841896911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-metrics-certs") pod "maas-api-59f6566bbc-gs27r" (UID: "0590fb3d-04f3-4064-a911-3fc95c92d7d1") : secret "maas-api-metrics-tls" not found
Sep 17 01:34:36.672010 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.671989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0590fb3d-04f3-4064-a911-3fc95c92d7d1-openshift-service-ca\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.673704 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.673679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-maas-api-tls\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:36.680568 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:36.680526    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls5gt\" (UniqueName: \"kubernetes.io/projected/0590fb3d-04f3-4064-a911-3fc95c92d7d1-kube-api-access-ls5gt\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:37.176891 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:37.176851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-metrics-certs\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:37.179453 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:37.179424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-metrics-certs\") pod \"maas-api-59f6566bbc-gs27r\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:37.407431 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:37.407392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:37.678933 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:37.678907    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"]
Sep 17 01:34:37.681231 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:34:37.681206    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0590fb3d_04f3_4064_a911_3fc95c92d7d1.slice/crio-175e01053bd1e338fa597f4b61164d49e679f6f44e0827f3444b1f26c2b151e2 WatchSource:0}: Error finding container 175e01053bd1e338fa597f4b61164d49e679f6f44e0827f3444b1f26c2b151e2: Status 404 returned error can't find the container with id 175e01053bd1e338fa597f4b61164d49e679f6f44e0827f3444b1f26c2b151e2
Sep 17 01:34:37.845159 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:37.845123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r" event={"ID":"0590fb3d-04f3-4064-a911-3fc95c92d7d1","Type":"ContainerStarted","Data":"175e01053bd1e338fa597f4b61164d49e679f6f44e0827f3444b1f26c2b151e2"}
Sep 17 01:34:39.856955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:39.856919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r" event={"ID":"0590fb3d-04f3-4064-a911-3fc95c92d7d1","Type":"ContainerStarted","Data":"79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628"}
Sep 17 01:34:39.857314 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:39.857045    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:39.873602 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:39.873556    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r" podStartSLOduration=2.164945691 podStartE2EDuration="3.873526326s" totalImagesPullingTime="1.708580635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:34:39.871076147 +0000 UTC m=+758.541592236" watchObservedRunningTime="2026-09-17 01:34:39.873526326 +0000 UTC m=+758.544042415"
Sep 17 01:34:45.866135 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:45.866107    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 01:34:53.972934 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:53.972900    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"]
Sep 17 01:34:53.976440 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:53.976421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:53.978798 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:53.978776    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 01:34:53.979724 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:53.979708    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 01:34:53.979853 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:53.979833    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 17 01:34:53.979940 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:53.979926    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6kzrq\""
Sep 17 01:34:53.985866 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:53.985845    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"]
Sep 17 01:34:54.024645 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.024614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.024772 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.024649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.024772 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.024667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdwwr\" (UniqueName: \"kubernetes.io/projected/1d5a37d2-95f2-40c4-9887-e65f49994bbb-kube-api-access-fdwwr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.024772 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.024689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.024872 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.024778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d5a37d2-95f2-40c4-9887-e65f49994bbb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.024872 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.024805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.125888 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.125857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d5a37d2-95f2-40c4-9887-e65f49994bbb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.125888 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.125892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.126094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.125932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.126094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.125954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.126094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.125990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdwwr\" (UniqueName: \"kubernetes.io/projected/1d5a37d2-95f2-40c4-9887-e65f49994bbb-kube-api-access-fdwwr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.126094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.126018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.126389 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.126359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.126507 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.126390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.126507 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.126432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.128168 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.128144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d5a37d2-95f2-40c4-9887-e65f49994bbb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.128505 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.128490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d5a37d2-95f2-40c4-9887-e65f49994bbb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.139123 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.139099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdwwr\" (UniqueName: \"kubernetes.io/projected/1d5a37d2-95f2-40c4-9887-e65f49994bbb-kube-api-access-fdwwr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc\" (UID: \"1d5a37d2-95f2-40c4-9887-e65f49994bbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.289134 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.289046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:34:54.560621 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.560436    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"]
Sep 17 01:34:54.563836 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:34:54.563798    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5a37d2_95f2_40c4_9887_e65f49994bbb.slice/crio-62499f1220ca2ef34d5d69a94aece7a301929fe4b9a270b50958ccdca2a92cdd WatchSource:0}: Error finding container 62499f1220ca2ef34d5d69a94aece7a301929fe4b9a270b50958ccdca2a92cdd: Status 404 returned error can't find the container with id 62499f1220ca2ef34d5d69a94aece7a301929fe4b9a270b50958ccdca2a92cdd
Sep 17 01:34:54.929069 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:54.928976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc" event={"ID":"1d5a37d2-95f2-40c4-9887-e65f49994bbb","Type":"ContainerStarted","Data":"62499f1220ca2ef34d5d69a94aece7a301929fe4b9a270b50958ccdca2a92cdd"}
Sep 17 01:34:55.934817 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:55.934778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc" event={"ID":"1d5a37d2-95f2-40c4-9887-e65f49994bbb","Type":"ContainerStarted","Data":"b73fdfa05e76f99a4f4c472c774e3c2b66a02a6755041e15905e145b91e3b73e"}
Sep 17 01:34:55.952684 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:34:55.952636    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc" podStartSLOduration=1.662852532 podStartE2EDuration="2.952617171s" totalImagesPullingTime="1.289764639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:34:55.949849991 +0000 UTC m=+774.620366079" watchObservedRunningTime="2026-09-17 01:34:55.952617171 +0000 UTC m=+774.623133266"
Sep 17 01:35:04.289175 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:04.289141    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:35:04.289635 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:04.289445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:35:04.302089 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:04.302063    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:35:04.984866 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:04.984839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw4jc"
Sep 17 01:35:15.254377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.254294    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"]
Sep 17 01:35:15.257871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.257846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.260099 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.260076    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 17 01:35:15.267360 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.267340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"]
Sep 17 01:35:15.319834 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.319804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.319834 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.319836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsf4w\" (UniqueName: \"kubernetes.io/projected/ed9383a0-b529-45a4-aabf-9ed9793ee947-kube-api-access-hsf4w\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.320019 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.319943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9383a0-b529-45a4-aabf-9ed9793ee947-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.320019 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.319978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.320091 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.320045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.320091 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.320062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.421002 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.420970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9383a0-b529-45a4-aabf-9ed9793ee947-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.421157 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.421009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.421157 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.421127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.421234 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.421160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.421234 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.421199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.421234 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.421226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsf4w\" (UniqueName: \"kubernetes.io/projected/ed9383a0-b529-45a4-aabf-9ed9793ee947-kube-api-access-hsf4w\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.421497 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.421467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.421643 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.421508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.421699 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.421649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.423252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.423232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9383a0-b529-45a4-aabf-9ed9793ee947-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.423529 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.423509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9383a0-b529-45a4-aabf-9ed9793ee947-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.437594 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.437571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsf4w\" (UniqueName: \"kubernetes.io/projected/ed9383a0-b529-45a4-aabf-9ed9793ee947-kube-api-access-hsf4w\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9\" (UID: \"ed9383a0-b529-45a4-aabf-9ed9793ee947\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.569203 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.569179    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:15.834837 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:15.834748    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"]
Sep 17 01:35:15.837067 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:35:15.837041    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9383a0_b529_45a4_aabf_9ed9793ee947.slice/crio-3065e2ec1de541e3a2e8ac0e06b5b335ba8c2fb98c851ae9443091ecfd9b09ed WatchSource:0}: Error finding container 3065e2ec1de541e3a2e8ac0e06b5b335ba8c2fb98c851ae9443091ecfd9b09ed: Status 404 returned error can't find the container with id 3065e2ec1de541e3a2e8ac0e06b5b335ba8c2fb98c851ae9443091ecfd9b09ed
Sep 17 01:35:16.022019 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:16.021985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9" event={"ID":"ed9383a0-b529-45a4-aabf-9ed9793ee947","Type":"ContainerStarted","Data":"3065e2ec1de541e3a2e8ac0e06b5b335ba8c2fb98c851ae9443091ecfd9b09ed"}
Sep 17 01:35:17.027887 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:17.027848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9" event={"ID":"ed9383a0-b529-45a4-aabf-9ed9793ee947","Type":"ContainerStarted","Data":"4e4a63fb47f449a295a586c202af36a6bad002ae7a1aa4c67a08d3a57df6d5cb"}
Sep 17 01:35:17.044138 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:17.044093    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9" podStartSLOduration=1.869465821 podStartE2EDuration="2.044077983s" totalImagesPullingTime="174.612162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:35:17.043075751 +0000 UTC m=+795.713591839" watchObservedRunningTime="2026-09-17 01:35:17.044077983 +0000 UTC m=+795.714594072"
Sep 17 01:35:25.569791 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:25.569697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:25.569791 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:25.569780    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:25.582226 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:25.582201    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:26.081110 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:26.081082    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rpqd9"
Sep 17 01:35:52.861953 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:52.861920    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"]
Sep 17 01:35:52.867521 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:52.867496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:52.869914 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:52.869882    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 17 01:35:52.873755 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:52.873728    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"]
Sep 17 01:35:52.959999 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:52.959959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:52.960195 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:52.960088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99rxk\" (UniqueName: \"kubernetes.io/projected/970b2ce0-c6e9-4175-9455-b8ec8e93423e-kube-api-access-99rxk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:52.960195 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:52.960163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:52.960195 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:52.960183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:52.960335 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:52.960204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:52.960335 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:52.960229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/970b2ce0-c6e9-4175-9455-b8ec8e93423e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.061666 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.061626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99rxk\" (UniqueName: \"kubernetes.io/projected/970b2ce0-c6e9-4175-9455-b8ec8e93423e-kube-api-access-99rxk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.061852 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.061787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.061852 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.061827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.061982 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.061865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.061982 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.061926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/970b2ce0-c6e9-4175-9455-b8ec8e93423e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.062088 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.061996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.062352 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.062246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.062352 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.062300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.062352 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.062340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.064285 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.064266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/970b2ce0-c6e9-4175-9455-b8ec8e93423e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.064471 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.064453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/970b2ce0-c6e9-4175-9455-b8ec8e93423e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.069177 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.069153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99rxk\" (UniqueName: \"kubernetes.io/projected/970b2ce0-c6e9-4175-9455-b8ec8e93423e-kube-api-access-99rxk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8\" (UID: \"970b2ce0-c6e9-4175-9455-b8ec8e93423e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.180710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.180609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:35:53.453611 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:53.453585    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"]
Sep 17 01:35:53.456354 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:35:53.456327    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970b2ce0_c6e9_4175_9455_b8ec8e93423e.slice/crio-ab8a333d838de39c79c3398c284644afe404a662e277c52f2af34705839235ec WatchSource:0}: Error finding container ab8a333d838de39c79c3398c284644afe404a662e277c52f2af34705839235ec: Status 404 returned error can't find the container with id ab8a333d838de39c79c3398c284644afe404a662e277c52f2af34705839235ec
Sep 17 01:35:54.201487 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:54.201449    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8" event={"ID":"970b2ce0-c6e9-4175-9455-b8ec8e93423e","Type":"ContainerStarted","Data":"7ca34eb142b0793f24161bdcd3e4ec6d199c50043cc1964d9aaa53619a281756"}
Sep 17 01:35:54.201896 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:54.201501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8" event={"ID":"970b2ce0-c6e9-4175-9455-b8ec8e93423e","Type":"ContainerStarted","Data":"ab8a333d838de39c79c3398c284644afe404a662e277c52f2af34705839235ec"}
Sep 17 01:35:54.218261 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:35:54.218203    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8" podStartSLOduration=2.043035495 podStartE2EDuration="2.218184686s" totalImagesPullingTime="175.149191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:35:54.216899309 +0000 UTC m=+832.887415422" watchObservedRunningTime="2026-09-17 01:35:54.218184686 +0000 UTC m=+832.888700774"
Sep 17 01:36:03.181164 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.181133    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:36:03.181647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.181232    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:36:03.193060 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.193036    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:36:03.254868 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.254841    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs55k8"
Sep 17 01:36:03.350482 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.350448    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"]
Sep 17 01:36:03.353840 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.353820    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.356136 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.356118    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 17 01:36:03.361314 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.361294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"]
Sep 17 01:36:03.463374 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.463272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.463374 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.463319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbedca61-0b53-4f9f-a7e4-af09699091f6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.463648 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.463415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.463648 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.463483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.463648 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.463505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.463648 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.463524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l26t\" (UniqueName: \"kubernetes.io/projected/bbedca61-0b53-4f9f-a7e4-af09699091f6-kube-api-access-9l26t\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.564071 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.564039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.564071 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.564073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.564303 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.564096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l26t\" (UniqueName: \"kubernetes.io/projected/bbedca61-0b53-4f9f-a7e4-af09699091f6-kube-api-access-9l26t\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.564303 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.564146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.564303 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.564175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbedca61-0b53-4f9f-a7e4-af09699091f6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.564303 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.564241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.564581 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.564558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.564663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.564614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.564663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.564631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.566275 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.566247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbedca61-0b53-4f9f-a7e4-af09699091f6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.566617 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.566598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbedca61-0b53-4f9f-a7e4-af09699091f6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.571141 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.571116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l26t\" (UniqueName: \"kubernetes.io/projected/bbedca61-0b53-4f9f-a7e4-af09699091f6-kube-api-access-9l26t\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx\" (UID: \"bbedca61-0b53-4f9f-a7e4-af09699091f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.663347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.663311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:03.929962 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:03.929937    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"]
Sep 17 01:36:03.931665 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:36:03.931639    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbedca61_0b53_4f9f_a7e4_af09699091f6.slice/crio-fafb40f6d855295a80b0acb98038759a3ac332525871bf2ff8c0e6f98af13648 WatchSource:0}: Error finding container fafb40f6d855295a80b0acb98038759a3ac332525871bf2ff8c0e6f98af13648: Status 404 returned error can't find the container with id fafb40f6d855295a80b0acb98038759a3ac332525871bf2ff8c0e6f98af13648
Sep 17 01:36:04.252996 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:04.252917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx" event={"ID":"bbedca61-0b53-4f9f-a7e4-af09699091f6","Type":"ContainerStarted","Data":"a5cbc05dedaf77821130073dddf87edc0ad15cb742bd546fb5939149a49419fd"}
Sep 17 01:36:04.252996 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:04.252969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx" event={"ID":"bbedca61-0b53-4f9f-a7e4-af09699091f6","Type":"ContainerStarted","Data":"fafb40f6d855295a80b0acb98038759a3ac332525871bf2ff8c0e6f98af13648"}
Sep 17 01:36:04.270883 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:04.270839    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx" podStartSLOduration=1.0912857 podStartE2EDuration="1.270823739s" totalImagesPullingTime="179.538039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:36:04.268117595 +0000 UTC m=+842.938633696" watchObservedRunningTime="2026-09-17 01:36:04.270823739 +0000 UTC m=+842.941339828"
Sep 17 01:36:13.663548 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:13.663499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:13.663932 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:13.663567    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:13.675989 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:13.675966    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:36:14.312292 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:36:14.312265    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kzbbx"
Sep 17 01:37:30.443640 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:30.443604    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69bcd7c777-xf8br"]
Sep 17 01:37:30.446903 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:30.446885    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69bcd7c777-xf8br"
Sep 17 01:37:30.455325 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:30.455284    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69bcd7c777-xf8br"]
Sep 17 01:37:30.475999 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:30.475957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0008dcc-a37c-4709-8c6e-e7eeb6916b9a-tls-cert\") pod \"authorino-69bcd7c777-xf8br\" (UID: \"b0008dcc-a37c-4709-8c6e-e7eeb6916b9a\") " pod="kuadrant-system/authorino-69bcd7c777-xf8br"
Sep 17 01:37:30.476153 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:30.476002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trwj\" (UniqueName: \"kubernetes.io/projected/b0008dcc-a37c-4709-8c6e-e7eeb6916b9a-kube-api-access-8trwj\") pod \"authorino-69bcd7c777-xf8br\" (UID: \"b0008dcc-a37c-4709-8c6e-e7eeb6916b9a\") " pod="kuadrant-system/authorino-69bcd7c777-xf8br"
Sep 17 01:37:30.576556 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:30.576498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0008dcc-a37c-4709-8c6e-e7eeb6916b9a-tls-cert\") pod \"authorino-69bcd7c777-xf8br\" (UID: \"b0008dcc-a37c-4709-8c6e-e7eeb6916b9a\") " pod="kuadrant-system/authorino-69bcd7c777-xf8br"
Sep 17 01:37:30.576734 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:30.576567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trwj\" (UniqueName: \"kubernetes.io/projected/b0008dcc-a37c-4709-8c6e-e7eeb6916b9a-kube-api-access-8trwj\") pod \"authorino-69bcd7c777-xf8br\" (UID: \"b0008dcc-a37c-4709-8c6e-e7eeb6916b9a\") " pod="kuadrant-system/authorino-69bcd7c777-xf8br"
Sep 17 01:37:30.579073 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:30.579052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0008dcc-a37c-4709-8c6e-e7eeb6916b9a-tls-cert\") pod \"authorino-69bcd7c777-xf8br\" (UID: \"b0008dcc-a37c-4709-8c6e-e7eeb6916b9a\") " pod="kuadrant-system/authorino-69bcd7c777-xf8br"
Sep 17 01:37:30.584066 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:30.584032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trwj\" (UniqueName: \"kubernetes.io/projected/b0008dcc-a37c-4709-8c6e-e7eeb6916b9a-kube-api-access-8trwj\") pod \"authorino-69bcd7c777-xf8br\" (UID: \"b0008dcc-a37c-4709-8c6e-e7eeb6916b9a\") " pod="kuadrant-system/authorino-69bcd7c777-xf8br"
Sep 17 01:37:30.756608 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:30.756496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69bcd7c777-xf8br"
Sep 17 01:37:31.024929 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:31.024860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69bcd7c777-xf8br"]
Sep 17 01:37:31.029512 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:37:31.029485    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0008dcc_a37c_4709_8c6e_e7eeb6916b9a.slice/crio-02574a5f02f644e82e3bdbdb1c5ee89ae86e0b4c7c18ac85a8d912fa425ad268 WatchSource:0}: Error finding container 02574a5f02f644e82e3bdbdb1c5ee89ae86e0b4c7c18ac85a8d912fa425ad268: Status 404 returned error can't find the container with id 02574a5f02f644e82e3bdbdb1c5ee89ae86e0b4c7c18ac85a8d912fa425ad268
Sep 17 01:37:31.667143 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:31.667105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69bcd7c777-xf8br" event={"ID":"b0008dcc-a37c-4709-8c6e-e7eeb6916b9a","Type":"ContainerStarted","Data":"02574a5f02f644e82e3bdbdb1c5ee89ae86e0b4c7c18ac85a8d912fa425ad268"}
Sep 17 01:37:32.674580 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:32.674512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69bcd7c777-xf8br" event={"ID":"b0008dcc-a37c-4709-8c6e-e7eeb6916b9a","Type":"ContainerStarted","Data":"80ada41f1ebda53718b55cca1bef7b5d3b3aa86256caa6ee2f747bca1ecf64d2"}
Sep 17 01:37:32.692834 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:32.692770    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-69bcd7c777-xf8br" podStartSLOduration=1.967819525 podStartE2EDuration="2.692748298s" totalImagesPullingTime="724.928773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:32.689595794 +0000 UTC m=+931.360111885" watchObservedRunningTime="2026-09-17 01:37:32.692748298 +0000 UTC m=+931.363264387"
Sep 17 01:37:32.720747 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:32.720712    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fc799b9bd-4s2nf"]
Sep 17 01:37:32.720973 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:32.720945    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5fc799b9bd-4s2nf" podUID="4c6b3327-a477-466e-8b07-6d4a3deda193" containerName="authorino" containerID="cri-o://49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99" gracePeriod=30
Sep 17 01:37:32.983025 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:32.982992    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc799b9bd-4s2nf"
Sep 17 01:37:32.994805 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:32.994761    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45qlf\" (UniqueName: \"kubernetes.io/projected/4c6b3327-a477-466e-8b07-6d4a3deda193-kube-api-access-45qlf\") pod \"4c6b3327-a477-466e-8b07-6d4a3deda193\" (UID: \"4c6b3327-a477-466e-8b07-6d4a3deda193\") "
Sep 17 01:37:32.994984 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:32.994843    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c6b3327-a477-466e-8b07-6d4a3deda193-tls-cert\") pod \"4c6b3327-a477-466e-8b07-6d4a3deda193\" (UID: \"4c6b3327-a477-466e-8b07-6d4a3deda193\") "
Sep 17 01:37:32.996942 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:32.996903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6b3327-a477-466e-8b07-6d4a3deda193-kube-api-access-45qlf" (OuterVolumeSpecName: "kube-api-access-45qlf") pod "4c6b3327-a477-466e-8b07-6d4a3deda193" (UID: "4c6b3327-a477-466e-8b07-6d4a3deda193"). InnerVolumeSpecName "kube-api-access-45qlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:37:33.009162 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.009121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6b3327-a477-466e-8b07-6d4a3deda193-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4c6b3327-a477-466e-8b07-6d4a3deda193" (UID: "4c6b3327-a477-466e-8b07-6d4a3deda193"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:37:33.095721 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.095686    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45qlf\" (UniqueName: \"kubernetes.io/projected/4c6b3327-a477-466e-8b07-6d4a3deda193-kube-api-access-45qlf\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:37:33.095721 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.095716    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c6b3327-a477-466e-8b07-6d4a3deda193-tls-cert\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:37:33.681701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.681668    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c6b3327-a477-466e-8b07-6d4a3deda193" containerID="49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99" exitCode=0
Sep 17 01:37:33.682094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.681713    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc799b9bd-4s2nf"
Sep 17 01:37:33.682094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.681756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc799b9bd-4s2nf" event={"ID":"4c6b3327-a477-466e-8b07-6d4a3deda193","Type":"ContainerDied","Data":"49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99"}
Sep 17 01:37:33.682094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.681790    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc799b9bd-4s2nf" event={"ID":"4c6b3327-a477-466e-8b07-6d4a3deda193","Type":"ContainerDied","Data":"037d53597c361e2630219dbddb0e8fd36c0254924d28a4aaa86dd4004aa2d20b"}
Sep 17 01:37:33.682094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.681811    2578 scope.go:117] "RemoveContainer" containerID="49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99"
Sep 17 01:37:33.691879 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.691857    2578 scope.go:117] "RemoveContainer" containerID="49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99"
Sep 17 01:37:33.692291 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:37:33.692264    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99\": container with ID starting with 49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99 not found: ID does not exist" containerID="49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99"
Sep 17 01:37:33.692373 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.692300    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99"} err="failed to get container status \"49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99\": rpc error: code = NotFound desc = could not find container \"49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99\": container with ID starting with 49be2e3ee2b9272062c946ef7730991e8dbefafea7863d4af7a1f2af76ddac99 not found: ID does not exist"
Sep 17 01:37:33.704701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.704670    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fc799b9bd-4s2nf"]
Sep 17 01:37:33.708176 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.708151    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fc799b9bd-4s2nf"]
Sep 17 01:37:33.920307 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:37:33.920269    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c6b3327-a477-466e-8b07-6d4a3deda193" path="/var/lib/kubelet/pods/4c6b3327-a477-466e-8b07-6d4a3deda193/volumes"
Sep 17 01:39:10.072375 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.072342    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"]
Sep 17 01:39:10.072859 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.072731    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c6b3327-a477-466e-8b07-6d4a3deda193" containerName="authorino"
Sep 17 01:39:10.072859 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.072744    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c6b3327-a477-466e-8b07-6d4a3deda193" containerName="authorino"
Sep 17 01:39:10.072859 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.072803    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c6b3327-a477-466e-8b07-6d4a3deda193" containerName="authorino"
Sep 17 01:39:10.075507 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.075484    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.077878 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.077853    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-8be734-serving-cert\""
Sep 17 01:39:10.087029 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.087005    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"]
Sep 17 01:39:10.222050 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.222017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzsrc\" (UniqueName: \"kubernetes.io/projected/7f500283-725c-4a50-8264-cce6604c0b7f-kube-api-access-mzsrc\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.222224 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.222085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.222224 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.222121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f500283-725c-4a50-8264-cce6604c0b7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.222224 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.222146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-metrics-certs\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.322939 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.322843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f500283-725c-4a50-8264-cce6604c0b7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.322939 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.322899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-metrics-certs\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.323174 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.322986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzsrc\" (UniqueName: \"kubernetes.io/projected/7f500283-725c-4a50-8264-cce6604c0b7f-kube-api-access-mzsrc\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.323174 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.323051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.323174 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:39:10.323165    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-serving-cert: secret "maas-api-e2e-worker-w0-8be734-serving-cert" not found
Sep 17 01:39:10.323336 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:39:10.323248    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-maas-api-tls podName:7f500283-725c-4a50-8264-cce6604c0b7f nodeName:}" failed. No retries permitted until 2026-09-17 01:39:10.823226534 +0000 UTC m=+1029.493742614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-maas-api-tls") pod "maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78" (UID: "7f500283-725c-4a50-8264-cce6604c0b7f") : secret "maas-api-e2e-worker-w0-8be734-serving-cert" not found
Sep 17 01:39:10.323608 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.323589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f500283-725c-4a50-8264-cce6604c0b7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.325610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.325586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-metrics-certs\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.333249 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.333226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzsrc\" (UniqueName: \"kubernetes.io/projected/7f500283-725c-4a50-8264-cce6604c0b7f-kube-api-access-mzsrc\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.827369 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.827337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.829780 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.829759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:10.987567 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:10.987508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:11.062562 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.062510    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"]
Sep 17 01:39:11.066407 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.066383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.069045 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.069025    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-cc98e0-serving-cert\""
Sep 17 01:39:11.078467 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.078341    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"]
Sep 17 01:39:11.230605 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.230570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.230772 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.230641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqvvr\" (UniqueName: \"kubernetes.io/projected/521b99ff-be70-4e0c-8464-e560b8e2fe46-kube-api-access-qqvvr\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.230772 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.230689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-metrics-certs\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.230772 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.230737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521b99ff-be70-4e0c-8464-e560b8e2fe46-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.261234 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.261204    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"]
Sep 17 01:39:11.263336 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:39:11.263296    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f500283_725c_4a50_8264_cce6604c0b7f.slice/crio-65c7baf8fef577d437f1155eb2505eaf535555a45a59eeb7e0ce79a0992d39ae WatchSource:0}: Error finding container 65c7baf8fef577d437f1155eb2505eaf535555a45a59eeb7e0ce79a0992d39ae: Status 404 returned error can't find the container with id 65c7baf8fef577d437f1155eb2505eaf535555a45a59eeb7e0ce79a0992d39ae
Sep 17 01:39:11.264711 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.264693    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 01:39:11.332076 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.331998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521b99ff-be70-4e0c-8464-e560b8e2fe46-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.332076 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.332063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.332289 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.332111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqvvr\" (UniqueName: \"kubernetes.io/projected/521b99ff-be70-4e0c-8464-e560b8e2fe46-kube-api-access-qqvvr\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.332289 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.332136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-metrics-certs\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.332289 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:39:11.332234    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-serving-cert: secret "maas-api-e2e-worker-w3-cc98e0-serving-cert" not found
Sep 17 01:39:11.332449 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:39:11.332331    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-maas-api-tls podName:521b99ff-be70-4e0c-8464-e560b8e2fe46 nodeName:}" failed. No retries permitted until 2026-09-17 01:39:11.832310372 +0000 UTC m=+1030.502826442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-maas-api-tls") pod "maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np" (UID: "521b99ff-be70-4e0c-8464-e560b8e2fe46") : secret "maas-api-e2e-worker-w3-cc98e0-serving-cert" not found
Sep 17 01:39:11.332825 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.332796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521b99ff-be70-4e0c-8464-e560b8e2fe46-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.334951 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.334931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-metrics-certs\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.345432 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.345413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqvvr\" (UniqueName: \"kubernetes.io/projected/521b99ff-be70-4e0c-8464-e560b8e2fe46-kube-api-access-qqvvr\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.837243 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.837209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.839731 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.839709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:11.959395 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.959308    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"]
Sep 17 01:39:11.963220 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.963196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:11.965444 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.965417    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d8d689-serving-cert\""
Sep 17 01:39:11.974672 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.974647    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"]
Sep 17 01:39:11.980802 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:11.980776    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:12.124924 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.124890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78" event={"ID":"7f500283-725c-4a50-8264-cce6604c0b7f","Type":"ContainerStarted","Data":"1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84"}
Sep 17 01:39:12.125299 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.124940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78" event={"ID":"7f500283-725c-4a50-8264-cce6604c0b7f","Type":"ContainerStarted","Data":"65c7baf8fef577d437f1155eb2505eaf535555a45a59eeb7e0ce79a0992d39ae"}
Sep 17 01:39:12.125299 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.125004    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:12.139772 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.139738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-metrics-certs\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.140038 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.139989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.140149 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.140063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-maas-api-tls\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.140442 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.140089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr65m\" (UniqueName: \"kubernetes.io/projected/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-kube-api-access-vr65m\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.141326 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.141283    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78" podStartSLOduration=1.813104235 podStartE2EDuration="2.141268534s" totalImagesPullingTime="328.164299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:12.139702855 +0000 UTC m=+1030.810218939" watchObservedRunningTime="2026-09-17 01:39:12.141268534 +0000 UTC m=+1030.811784622"
Sep 17 01:39:12.242090 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.242011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.242217 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.242113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-maas-api-tls\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.242217 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.242150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr65m\" (UniqueName: \"kubernetes.io/projected/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-kube-api-access-vr65m\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.242331 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:39:12.242287    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-serving-cert: secret "maas-api-e2e-inf-a-d8d689-serving-cert" not found
Sep 17 01:39:12.242331 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.242310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-metrics-certs\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.242429 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:39:12.242360    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-maas-api-tls podName:6d92afd7-06f7-4ad4-b8b2-9233fcfd3154 nodeName:}" failed. No retries permitted until 2026-09-17 01:39:12.742337774 +0000 UTC m=+1031.412853856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-maas-api-tls") pod "maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486" (UID: "6d92afd7-06f7-4ad4-b8b2-9233fcfd3154") : secret "maas-api-e2e-inf-a-d8d689-serving-cert" not found
Sep 17 01:39:12.243077 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.243053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.244834 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.244811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-metrics-certs\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.252136 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.252109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr65m\" (UniqueName: \"kubernetes.io/projected/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-kube-api-access-vr65m\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.261131 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.261110    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"]
Sep 17 01:39:12.262778 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:39:12.262754    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521b99ff_be70_4e0c_8464_e560b8e2fe46.slice/crio-70da8896260c40d9b7737916ea8cfac33076bbde9145477708f4bea0ee477a62 WatchSource:0}: Error finding container 70da8896260c40d9b7737916ea8cfac33076bbde9145477708f4bea0ee477a62: Status 404 returned error can't find the container with id 70da8896260c40d9b7737916ea8cfac33076bbde9145477708f4bea0ee477a62
Sep 17 01:39:12.748672 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.748634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-maas-api-tls\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.751069 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.751046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-maas-api-tls\") pod \"maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:12.877596 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:12.877485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:13.131692 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:13.131594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np" event={"ID":"521b99ff-be70-4e0c-8464-e560b8e2fe46","Type":"ContainerStarted","Data":"b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f"}
Sep 17 01:39:13.131692 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:13.131668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np" event={"ID":"521b99ff-be70-4e0c-8464-e560b8e2fe46","Type":"ContainerStarted","Data":"70da8896260c40d9b7737916ea8cfac33076bbde9145477708f4bea0ee477a62"}
Sep 17 01:39:13.132278 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:13.131887    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:13.152587 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:13.152513    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np" podStartSLOduration=1.8149972490000001 podStartE2EDuration="2.15249256s" totalImagesPullingTime="337.495311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:13.150615561 +0000 UTC m=+1031.821131650" watchObservedRunningTime="2026-09-17 01:39:13.15249256 +0000 UTC m=+1031.823008650"
Sep 17 01:39:13.155961 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:39:13.155940    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d92afd7_06f7_4ad4_b8b2_9233fcfd3154.slice/crio-3e12cc5820e31fc3d7193b917ee45c530e5311c3ac7e27e48763754e5c43cd4b WatchSource:0}: Error finding container 3e12cc5820e31fc3d7193b917ee45c530e5311c3ac7e27e48763754e5c43cd4b: Status 404 returned error can't find the container with id 3e12cc5820e31fc3d7193b917ee45c530e5311c3ac7e27e48763754e5c43cd4b
Sep 17 01:39:13.156138 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:13.156118    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"]
Sep 17 01:39:14.138773 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:14.138737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486" event={"ID":"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154","Type":"ContainerStarted","Data":"4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb"}
Sep 17 01:39:14.139247 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:14.138998    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:14.139247 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:14.139047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486" event={"ID":"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154","Type":"ContainerStarted","Data":"3e12cc5820e31fc3d7193b917ee45c530e5311c3ac7e27e48763754e5c43cd4b"}
Sep 17 01:39:14.155741 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:14.155696    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486" podStartSLOduration=2.832450595 podStartE2EDuration="3.155682437s" totalImagesPullingTime="323.231842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:14.15390647 +0000 UTC m=+1032.824422570" watchObservedRunningTime="2026-09-17 01:39:14.155682437 +0000 UTC m=+1032.826198569"
Sep 17 01:39:18.137402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:18.137376    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:39:19.144706 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:19.144679    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:39:20.149715 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:20.149684    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:39:44.360851 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.360808    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"]
Sep 17 01:39:44.364625 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.364604    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.368082 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.368061    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8be734\"/\"openshift-service-ca.crt\""
Sep 17 01:39:44.368208 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.368093    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8be734\"/\"default-dockercfg-fx72h\""
Sep 17 01:39:44.368208 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.368093    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8be734\"/\"kube-root-ca.crt\""
Sep 17 01:39:44.368208 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.368062    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8be734\"/\"simulator-w0-8be734-kserve-self-signed-certs\""
Sep 17 01:39:44.374124 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.374100    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"]
Sep 17 01:39:44.436184 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.436148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84d8s\" (UniqueName: \"kubernetes.io/projected/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-kube-api-access-84d8s\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.436377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.436243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tls-certs\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.436377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.436289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-model-cache\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.436377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.436363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tmp-dir\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.436593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.436432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-home\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.436593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.436473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-dshm\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.537233 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.537180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tls-certs\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.537233 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.537236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-model-cache\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.537470 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.537277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tmp-dir\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.537470 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.537337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-home\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.537470 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.537372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-dshm\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.537470 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.537417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8s\" (UniqueName: \"kubernetes.io/projected/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-kube-api-access-84d8s\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.537779 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.537750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-model-cache\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.537844 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.537797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tmp-dir\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.537886 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.537859    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-home\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.539517 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.539496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-dshm\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.539681 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.539664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tls-certs\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.544222 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.544199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84d8s\" (UniqueName: \"kubernetes.io/projected/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-kube-api-access-84d8s\") pod \"simulator-w0-8be734-kserve-b477c4bf4-7dt9c\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") " pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.674128 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.674041    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:44.945820 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:44.945790    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"]
Sep 17 01:39:44.948297 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:39:44.948258    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97a445cd_d987_46bb_b3b7_e5495d1ebb3d.slice/crio-428bce6661ef8521855e7f585073f1d1beed38e3bbf4ab9468be82cf531ca45e WatchSource:0}: Error finding container 428bce6661ef8521855e7f585073f1d1beed38e3bbf4ab9468be82cf531ca45e: Status 404 returned error can't find the container with id 428bce6661ef8521855e7f585073f1d1beed38e3bbf4ab9468be82cf531ca45e
Sep 17 01:39:45.298801 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.298769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c" event={"ID":"97a445cd-d987-46bb-b3b7-e5495d1ebb3d","Type":"ContainerStarted","Data":"2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189"}
Sep 17 01:39:45.298969 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.298806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c" event={"ID":"97a445cd-d987-46bb-b3b7-e5495d1ebb3d","Type":"ContainerStarted","Data":"428bce6661ef8521855e7f585073f1d1beed38e3bbf4ab9468be82cf531ca45e"}
Sep 17 01:39:45.314039 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.313994    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c" podStartSLOduration=1.3139795 podStartE2EDuration="1.3139795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:45.312845093 +0000 UTC m=+1063.983361181" watchObservedRunningTime="2026-09-17 01:39:45.3139795 +0000 UTC m=+1063.984495588"
Sep 17 01:39:45.582898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.582821    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"]
Sep 17 01:39:45.588083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.588059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.590252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.590232    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d370b3-serving-cert\""
Sep 17 01:39:45.598334 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.598309    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"]
Sep 17 01:39:45.649445 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.649406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9dvk\" (UniqueName: \"kubernetes.io/projected/9ae8a13e-21f0-46d5-964b-6b164df2a269-kube-api-access-b9dvk\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.649614 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.649457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-maas-api-tls\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.649614 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.649485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae8a13e-21f0-46d5-964b-6b164df2a269-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.649614 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.649518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-metrics-certs\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.751106 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.751073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9dvk\" (UniqueName: \"kubernetes.io/projected/9ae8a13e-21f0-46d5-964b-6b164df2a269-kube-api-access-b9dvk\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.751270 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.751114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-maas-api-tls\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.751270 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.751142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae8a13e-21f0-46d5-964b-6b164df2a269-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.751270 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.751175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-metrics-certs\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.751270 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:39:45.751211    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-serving-cert: secret "maas-api-e2e-inf-b-d370b3-serving-cert" not found
Sep 17 01:39:45.751461 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:39:45.751280    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-maas-api-tls podName:9ae8a13e-21f0-46d5-964b-6b164df2a269 nodeName:}" failed. No retries permitted until 2026-09-17 01:39:46.251260002 +0000 UTC m=+1064.921776069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-maas-api-tls") pod "maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67" (UID: "9ae8a13e-21f0-46d5-964b-6b164df2a269") : secret "maas-api-e2e-inf-b-d370b3-serving-cert" not found
Sep 17 01:39:45.751854 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.751835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae8a13e-21f0-46d5-964b-6b164df2a269-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.753904 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.753879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-metrics-certs\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:45.758712 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:45.758692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9dvk\" (UniqueName: \"kubernetes.io/projected/9ae8a13e-21f0-46d5-964b-6b164df2a269-kube-api-access-b9dvk\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:46.257009 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:46.256975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-maas-api-tls\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:46.259317 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:46.259299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-maas-api-tls\") pod \"maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:46.502612 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:46.502581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:46.773995 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:46.773960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"]
Sep 17 01:39:46.775548 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:39:46.775506    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae8a13e_21f0_46d5_964b_6b164df2a269.slice/crio-36e786545c130c870bf7023fe6d45152c000477b21e0d4ab0ac654a086e69b09 WatchSource:0}: Error finding container 36e786545c130c870bf7023fe6d45152c000477b21e0d4ab0ac654a086e69b09: Status 404 returned error can't find the container with id 36e786545c130c870bf7023fe6d45152c000477b21e0d4ab0ac654a086e69b09
Sep 17 01:39:47.309624 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:47.309587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67" event={"ID":"9ae8a13e-21f0-46d5-964b-6b164df2a269","Type":"ContainerStarted","Data":"b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28"}
Sep 17 01:39:47.309774 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:47.309663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67" event={"ID":"9ae8a13e-21f0-46d5-964b-6b164df2a269","Type":"ContainerStarted","Data":"36e786545c130c870bf7023fe6d45152c000477b21e0d4ab0ac654a086e69b09"}
Sep 17 01:39:47.309774 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:47.309745    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:47.327586 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:47.327513    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67" podStartSLOduration=1.949507664 podStartE2EDuration="2.327499645s" totalImagesPullingTime="377.991981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:47.325405933 +0000 UTC m=+1065.995922012" watchObservedRunningTime="2026-09-17 01:39:47.327499645 +0000 UTC m=+1065.998015793"
Sep 17 01:39:53.319767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:53.319740    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:39:54.675143 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:54.675061    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:54.675143 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:54.675113    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:54.687621 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:54.687595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:39:55.359237 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:39:55.359209    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:40:15.551118 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.551079    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"]
Sep 17 01:40:15.554777 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.554761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.557206 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.557185    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cc98e0\"/\"openshift-service-ca.crt\""
Sep 17 01:40:15.557330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.557227    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cc98e0\"/\"kube-root-ca.crt\""
Sep 17 01:40:15.558169 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.558146    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cc98e0\"/\"simulator-w3-cc98e0-kserve-self-signed-certs\""
Sep 17 01:40:15.558301 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.558168    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cc98e0\"/\"default-dockercfg-tv6kc\""
Sep 17 01:40:15.564041 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.564020    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"]
Sep 17 01:40:15.637160 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.637119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-dshm\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.637347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.637183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14053846-72b9-4e45-987d-a26675e0ebb8-tls-certs\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.637347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.637293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-tmp-dir\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.637347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.637333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8kbm\" (UniqueName: \"kubernetes.io/projected/14053846-72b9-4e45-987d-a26675e0ebb8-kube-api-access-r8kbm\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.637519 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.637407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-home\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.637519 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.637469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-model-cache\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.738345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.738304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-model-cache\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.738518 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.738363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-dshm\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.738518 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.738392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14053846-72b9-4e45-987d-a26675e0ebb8-tls-certs\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.738518 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.738408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-tmp-dir\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.738518 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.738426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8kbm\" (UniqueName: \"kubernetes.io/projected/14053846-72b9-4e45-987d-a26675e0ebb8-kube-api-access-r8kbm\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.738518 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.738469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-home\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.738854 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.738830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-model-cache\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.738904 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.738857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-tmp-dir\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.738941 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.738892    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-home\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.740727 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.740702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-dshm\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.740995 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.740976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14053846-72b9-4e45-987d-a26675e0ebb8-tls-certs\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.745807 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.745782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8kbm\" (UniqueName: \"kubernetes.io/projected/14053846-72b9-4e45-987d-a26675e0ebb8-kube-api-access-r8kbm\") pod \"simulator-w3-cc98e0-kserve-68756b55c4-6kf78\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") " pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:15.864931 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:15.864837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:16.136406 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:16.136381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"]
Sep 17 01:40:16.138246 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:40:16.138211    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14053846_72b9_4e45_987d_a26675e0ebb8.slice/crio-11bc07e8ec4abcdfae0880a823556ce042e45cc90f91541bb62c24b2a91e09ce WatchSource:0}: Error finding container 11bc07e8ec4abcdfae0880a823556ce042e45cc90f91541bb62c24b2a91e09ce: Status 404 returned error can't find the container with id 11bc07e8ec4abcdfae0880a823556ce042e45cc90f91541bb62c24b2a91e09ce
Sep 17 01:40:16.449032 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:16.448942    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78" event={"ID":"14053846-72b9-4e45-987d-a26675e0ebb8","Type":"ContainerStarted","Data":"78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6"}
Sep 17 01:40:16.449032 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:16.448984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78" event={"ID":"14053846-72b9-4e45-987d-a26675e0ebb8","Type":"ContainerStarted","Data":"11bc07e8ec4abcdfae0880a823556ce042e45cc90f91541bb62c24b2a91e09ce"}
Sep 17 01:40:16.463808 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:16.463758    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78" podStartSLOduration=1.463743706 podStartE2EDuration="1.463743706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:40:16.463462436 +0000 UTC m=+1095.133978525" watchObservedRunningTime="2026-09-17 01:40:16.463743706 +0000 UTC m=+1095.134259794"
Sep 17 01:40:25.865152 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:25.865103    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:25.865579 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:25.865174    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:25.877131 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:25.877109    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:26.509675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:26.509637    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:40:52.572447 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.572411    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"]
Sep 17 01:40:52.576199 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.576174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.578478 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.578439    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d8d689\"/\"default-dockercfg-hjwb5\""
Sep 17 01:40:52.578727 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.578705    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d8d689\"/\"openshift-service-ca.crt\""
Sep 17 01:40:52.579422 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.579404    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d8d689\"/\"test-model-d8d689-kserve-self-signed-certs\""
Sep 17 01:40:52.579523 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.579450    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d8d689\"/\"kube-root-ca.crt\""
Sep 17 01:40:52.584481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.584138    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"]
Sep 17 01:40:52.698832 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.698796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tmp-dir\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.699034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.698845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-dshm\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.699034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.698918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-model-cache\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.699034 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.698979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm4jd\" (UniqueName: \"kubernetes.io/projected/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-kube-api-access-vm4jd\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.699177 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.699044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-home\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.699177 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.699067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tls-certs\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.800524 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.800472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-model-cache\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.800739 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.800564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm4jd\" (UniqueName: \"kubernetes.io/projected/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-kube-api-access-vm4jd\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.800739 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.800615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-home\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.800739 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.800648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tls-certs\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.800739 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.800696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tmp-dir\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.800944 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.800745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-dshm\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.801007 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.800968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-model-cache\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.801063 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.801021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-home\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.801150 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.801127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tmp-dir\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.803015 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.802996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-dshm\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.803300 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.803281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tls-certs\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.807923 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.807900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm4jd\" (UniqueName: \"kubernetes.io/projected/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-kube-api-access-vm4jd\") pod \"test-model-d8d689-kserve-7996db9d4c-2fqpn\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") " pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:52.887617 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:52.887490    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:40:53.155471 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:53.155446    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"]
Sep 17 01:40:53.157703 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:40:53.157671    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00a42ab1_27ab_4f2d_8774_cb1d536a94fc.slice/crio-78098e4b102b117b7a3e717f19e9db8e0e8ff2b927e1605632bcc1dfb2bcbbd7 WatchSource:0}: Error finding container 78098e4b102b117b7a3e717f19e9db8e0e8ff2b927e1605632bcc1dfb2bcbbd7: Status 404 returned error can't find the container with id 78098e4b102b117b7a3e717f19e9db8e0e8ff2b927e1605632bcc1dfb2bcbbd7
Sep 17 01:40:53.627340 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:53.627304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn" event={"ID":"00a42ab1-27ab-4f2d-8774-cb1d536a94fc","Type":"ContainerStarted","Data":"4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef"}
Sep 17 01:40:53.627340 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:53.627347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn" event={"ID":"00a42ab1-27ab-4f2d-8774-cb1d536a94fc","Type":"ContainerStarted","Data":"78098e4b102b117b7a3e717f19e9db8e0e8ff2b927e1605632bcc1dfb2bcbbd7"}
Sep 17 01:40:53.643021 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:40:53.642975    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn" podStartSLOduration=1.642959035 podStartE2EDuration="1.642959035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:40:53.641827613 +0000 UTC m=+1132.312343703" watchObservedRunningTime="2026-09-17 01:40:53.642959035 +0000 UTC m=+1132.313475124"
Sep 17 01:41:02.888384 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:41:02.888340    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:41:02.888773 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:41:02.888405    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:41:02.901122 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:41:02.901087    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:41:03.687826 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:41:03.687798    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:42:58.872461 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:58.872371    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"]
Sep 17 01:42:58.876178 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:58.876156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:58.878857 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:58.878835    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cc98e0\"/\"distinct-w3-cc98e0-kserve-self-signed-certs\""
Sep 17 01:42:58.885212 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:58.885184    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"]
Sep 17 01:42:58.967188 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:58.967149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6mkw\" (UniqueName: \"kubernetes.io/projected/c6f37728-182c-44b2-a837-8f772236f67b-kube-api-access-c6mkw\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:58.967391 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:58.967279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-model-cache\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:58.967391 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:58.967343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-dshm\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:58.967391 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:58.967366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-home\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:58.967391 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:58.967380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-tmp-dir\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:58.967610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:58.967481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f37728-182c-44b2-a837-8f772236f67b-tls-certs\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.069031 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.068987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-model-cache\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.069219 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.069098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-dshm\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.069219 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.069132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-home\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.069219 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.069155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-tmp-dir\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.069380 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.069238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f37728-182c-44b2-a837-8f772236f67b-tls-certs\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.069380 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.069296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6mkw\" (UniqueName: \"kubernetes.io/projected/c6f37728-182c-44b2-a837-8f772236f67b-kube-api-access-c6mkw\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.069491 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.069442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-model-cache\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.069570 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.069559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-tmp-dir\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.069765 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.069744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-home\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.071489 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.071464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-dshm\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.071945 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.071924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f37728-182c-44b2-a837-8f772236f67b-tls-certs\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.076781 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.076763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6mkw\" (UniqueName: \"kubernetes.io/projected/c6f37728-182c-44b2-a837-8f772236f67b-kube-api-access-c6mkw\") pod \"distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.187136 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.187038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:42:59.458871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:42:59.458840    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"]
Sep 17 01:42:59.460010 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:42:59.459983    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6f37728_182c_44b2_a837_8f772236f67b.slice/crio-903c1be40b27b6dad659ba4ab29c0d59796176f562f46c1d16f846d7c64da998 WatchSource:0}: Error finding container 903c1be40b27b6dad659ba4ab29c0d59796176f562f46c1d16f846d7c64da998: Status 404 returned error can't find the container with id 903c1be40b27b6dad659ba4ab29c0d59796176f562f46c1d16f846d7c64da998
Sep 17 01:43:00.254804 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:00.254771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf" event={"ID":"c6f37728-182c-44b2-a837-8f772236f67b","Type":"ContainerStarted","Data":"4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4"}
Sep 17 01:43:00.254804 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:00.254813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf" event={"ID":"c6f37728-182c-44b2-a837-8f772236f67b","Type":"ContainerStarted","Data":"903c1be40b27b6dad659ba4ab29c0d59796176f562f46c1d16f846d7c64da998"}
Sep 17 01:43:00.269469 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:00.269420    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf" podStartSLOduration=2.269402535 podStartE2EDuration="2.269402535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:43:00.268627285 +0000 UTC m=+1258.939143374" watchObservedRunningTime="2026-09-17 01:43:00.269402535 +0000 UTC m=+1258.939918623"
Sep 17 01:43:09.187952 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:09.187914    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:43:09.188407 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:09.187972    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:43:09.200583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:09.200530    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:43:09.311421 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:09.311394    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:43:56.444550 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.444491    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"]
Sep 17 01:43:56.448520 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.448495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.452504 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.452482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cc98e0\"/\"distinct-2-w3-cc98e0-kserve-self-signed-certs\""
Sep 17 01:43:56.459429 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.459402    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"]
Sep 17 01:43:56.554812 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.554769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb4zt\" (UniqueName: \"kubernetes.io/projected/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-kube-api-access-sb4zt\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.554995 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.554843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-dshm\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.554995 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.554873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tmp-dir\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.555071 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.555009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-model-cache\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.555071 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.555034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tls-certs\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.555147 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.555116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-home\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.656484 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.656443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tmp-dir\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.656681 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.656518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-model-cache\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.656681 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.656563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tls-certs\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.656756 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.656708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-home\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.656796 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.656782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4zt\" (UniqueName: \"kubernetes.io/projected/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-kube-api-access-sb4zt\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.656831 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.656822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-dshm\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.656955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.656932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tmp-dir\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.657018 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.656955    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-model-cache\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.657184 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.657154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-home\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.659098 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.659079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-dshm\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.659183 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.659168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tls-certs\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.675598 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.675569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb4zt\" (UniqueName: \"kubernetes.io/projected/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-kube-api-access-sb4zt\") pod \"distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") " pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:56.760914 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:56.760827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:43:57.031781 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:57.031757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"]
Sep 17 01:43:57.034000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:43:57.033976    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9074d984_88b4_4b9f_a2ce_eb9aeb50a4b7.slice/crio-fd3009874c303a87fdf8f392d1e6965c8f39c76de6d958c5d834e5578dbce55b WatchSource:0}: Error finding container fd3009874c303a87fdf8f392d1e6965c8f39c76de6d958c5d834e5578dbce55b: Status 404 returned error can't find the container with id fd3009874c303a87fdf8f392d1e6965c8f39c76de6d958c5d834e5578dbce55b
Sep 17 01:43:57.533725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:57.533686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj" event={"ID":"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7","Type":"ContainerStarted","Data":"4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c"}
Sep 17 01:43:57.534157 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:57.533735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj" event={"ID":"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7","Type":"ContainerStarted","Data":"fd3009874c303a87fdf8f392d1e6965c8f39c76de6d958c5d834e5578dbce55b"}
Sep 17 01:43:57.550230 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:57.550170    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj" podStartSLOduration=1.5501487699999998 podStartE2EDuration="1.55014877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:43:57.549297478 +0000 UTC m=+1316.219813571" watchObservedRunningTime="2026-09-17 01:43:57.55014877 +0000 UTC m=+1316.220664860"
Sep 17 01:43:59.953014 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:43:59.952982    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"]
Sep 17 01:44:06.761765 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:06.761723    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:44:06.762249 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:06.761780    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:44:06.774437 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:06.774412    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:44:07.594926 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:07.594895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:44:14.972598 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:14.972552    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn" podUID="00a42ab1-27ab-4f2d-8774-cb1d536a94fc" containerName="main" containerID="cri-o://4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef" gracePeriod=45
Sep 17 01:44:15.224836 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.224806    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:44:15.255103 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tls-certs\") pod \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") "
Sep 17 01:44:15.255283 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255127    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-home\") pod \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") "
Sep 17 01:44:15.255283 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255232    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tmp-dir\") pod \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") "
Sep 17 01:44:15.255283 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-dshm\") pod \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") "
Sep 17 01:44:15.255460 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255366    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-model-cache\") pod \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") "
Sep 17 01:44:15.255460 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255387    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-home" (OuterVolumeSpecName: "home") pod "00a42ab1-27ab-4f2d-8774-cb1d536a94fc" (UID: "00a42ab1-27ab-4f2d-8774-cb1d536a94fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:44:15.255601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255489    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00a42ab1-27ab-4f2d-8774-cb1d536a94fc" (UID: "00a42ab1-27ab-4f2d-8774-cb1d536a94fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:44:15.255601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255513    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm4jd\" (UniqueName: \"kubernetes.io/projected/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-kube-api-access-vm4jd\") pod \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\" (UID: \"00a42ab1-27ab-4f2d-8774-cb1d536a94fc\") "
Sep 17 01:44:15.255601 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255581    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "00a42ab1-27ab-4f2d-8774-cb1d536a94fc" (UID: "00a42ab1-27ab-4f2d-8774-cb1d536a94fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:44:15.255891 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255873    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:15.255954 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255895    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:15.255954 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.255908    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:15.257423 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.257395    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-dshm" (OuterVolumeSpecName: "dshm") pod "00a42ab1-27ab-4f2d-8774-cb1d536a94fc" (UID: "00a42ab1-27ab-4f2d-8774-cb1d536a94fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:44:15.257423 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.257415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00a42ab1-27ab-4f2d-8774-cb1d536a94fc" (UID: "00a42ab1-27ab-4f2d-8774-cb1d536a94fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:44:15.257886 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.257864    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-kube-api-access-vm4jd" (OuterVolumeSpecName: "kube-api-access-vm4jd") pod "00a42ab1-27ab-4f2d-8774-cb1d536a94fc" (UID: "00a42ab1-27ab-4f2d-8774-cb1d536a94fc"). InnerVolumeSpecName "kube-api-access-vm4jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:44:15.356454 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.356375    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm4jd\" (UniqueName: \"kubernetes.io/projected/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-kube-api-access-vm4jd\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:15.356454 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.356403    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:15.356454 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.356413    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a42ab1-27ab-4f2d-8774-cb1d536a94fc-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:15.624843 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.624758    2578 generic.go:356] "Generic (PLEG): container finished" podID="00a42ab1-27ab-4f2d-8774-cb1d536a94fc" containerID="4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef" exitCode=0
Sep 17 01:44:15.624843 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.624815    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"
Sep 17 01:44:15.625057 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.624878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn" event={"ID":"00a42ab1-27ab-4f2d-8774-cb1d536a94fc","Type":"ContainerDied","Data":"4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef"}
Sep 17 01:44:15.625057 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.624917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn" event={"ID":"00a42ab1-27ab-4f2d-8774-cb1d536a94fc","Type":"ContainerDied","Data":"78098e4b102b117b7a3e717f19e9db8e0e8ff2b927e1605632bcc1dfb2bcbbd7"}
Sep 17 01:44:15.625057 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.624936    2578 scope.go:117] "RemoveContainer" containerID="4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef"
Sep 17 01:44:15.633940 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.633920    2578 scope.go:117] "RemoveContainer" containerID="4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef"
Sep 17 01:44:15.634200 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:44:15.634180    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef\": container with ID starting with 4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef not found: ID does not exist" containerID="4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef"
Sep 17 01:44:15.634267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.634208    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef"} err="failed to get container status \"4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef\": rpc error: code = NotFound desc = could not find container \"4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef\": container with ID starting with 4d74a35afe4b33127a10cb76f8a84d22184d86e8fed81e80e589d7fe429c92ef not found: ID does not exist"
Sep 17 01:44:15.649212 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.649190    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"]
Sep 17 01:44:15.655659 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.655636    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d8d689/test-model-d8d689-kserve-7996db9d4c-2fqpn"]
Sep 17 01:44:15.659930 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:44:15.659906    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00a42ab1_27ab_4f2d_8774_cb1d536a94fc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00a42ab1_27ab_4f2d_8774_cb1d536a94fc.slice/crio-78098e4b102b117b7a3e717f19e9db8e0e8ff2b927e1605632bcc1dfb2bcbbd7\": RecentStats: unable to find data in memory cache]"
Sep 17 01:44:15.920899 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:15.920820    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00a42ab1-27ab-4f2d-8774-cb1d536a94fc" path="/var/lib/kubelet/pods/00a42ab1-27ab-4f2d-8774-cb1d536a94fc/volumes"
Sep 17 01:44:35.848028 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.847946    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"]
Sep 17 01:44:35.848643 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.848425    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a42ab1-27ab-4f2d-8774-cb1d536a94fc" containerName="main"
Sep 17 01:44:35.848643 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.848440    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a42ab1-27ab-4f2d-8774-cb1d536a94fc" containerName="main"
Sep 17 01:44:35.848643 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.848524    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a42ab1-27ab-4f2d-8774-cb1d536a94fc" containerName="main"
Sep 17 01:44:35.851722 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.851707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:35.854052 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.854033    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8be734\"/\"distinct-2-w0-8be734-kserve-self-signed-certs\""
Sep 17 01:44:35.861314 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.861293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"]
Sep 17 01:44:35.966758 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.966728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a55a27-78c1-4cf7-ba82-00034f830d99-tls-certs\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:35.966758 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.966763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-model-cache\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:35.966967 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.966838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-dshm\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:35.966967 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.966867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwvv\" (UniqueName: \"kubernetes.io/projected/94a55a27-78c1-4cf7-ba82-00034f830d99-kube-api-access-ftwvv\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:35.966967 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.966950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-tmp-dir\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:35.967067 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:35.966985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-home\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.068023 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.067987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-dshm\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.068023 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.068027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwvv\" (UniqueName: \"kubernetes.io/projected/94a55a27-78c1-4cf7-ba82-00034f830d99-kube-api-access-ftwvv\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.068288 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.068107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-tmp-dir\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.068288 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.068144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-home\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.068415 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.068296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a55a27-78c1-4cf7-ba82-00034f830d99-tls-certs\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.068415 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.068338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-model-cache\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.068667 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.068633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-tmp-dir\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.068815 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.068665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-home\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.068815 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.068801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-model-cache\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.070637 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.070617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-dshm\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.070890 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.070872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a55a27-78c1-4cf7-ba82-00034f830d99-tls-certs\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.074984 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.074966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwvv\" (UniqueName: \"kubernetes.io/projected/94a55a27-78c1-4cf7-ba82-00034f830d99-kube-api-access-ftwvv\") pod \"distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") " pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.161631 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.161552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:36.432600 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.432572    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"]
Sep 17 01:44:36.435103 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:44:36.435076    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94a55a27_78c1_4cf7_ba82_00034f830d99.slice/crio-7647be581d5c94ede1c5ec25ee9a96eaa25524effae49803ab0e7ec7071289bb WatchSource:0}: Error finding container 7647be581d5c94ede1c5ec25ee9a96eaa25524effae49803ab0e7ec7071289bb: Status 404 returned error can't find the container with id 7647be581d5c94ede1c5ec25ee9a96eaa25524effae49803ab0e7ec7071289bb
Sep 17 01:44:36.734146 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.734055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw" event={"ID":"94a55a27-78c1-4cf7-ba82-00034f830d99","Type":"ContainerStarted","Data":"3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0"}
Sep 17 01:44:36.734146 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.734101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw" event={"ID":"94a55a27-78c1-4cf7-ba82-00034f830d99","Type":"ContainerStarted","Data":"7647be581d5c94ede1c5ec25ee9a96eaa25524effae49803ab0e7ec7071289bb"}
Sep 17 01:44:36.749892 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:36.749843    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw" podStartSLOduration=1.749826882 podStartE2EDuration="1.749826882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:44:36.747801971 +0000 UTC m=+1355.418318094" watchObservedRunningTime="2026-09-17 01:44:36.749826882 +0000 UTC m=+1355.420343037"
Sep 17 01:44:37.002895 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.002814    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"]
Sep 17 01:44:37.003329 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.003073    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486" podUID="6d92afd7-06f7-4ad4-b8b2-9233fcfd3154" containerName="maas-api" containerID="cri-o://4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb" gracePeriod=30
Sep 17 01:44:37.247902 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.247873    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:44:37.380503 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.380473    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-maas-api-tls\") pod \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") "
Sep 17 01:44:37.380718 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.380601    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr65m\" (UniqueName: \"kubernetes.io/projected/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-kube-api-access-vr65m\") pod \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") "
Sep 17 01:44:37.380718 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.380625    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-metrics-certs\") pod \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") "
Sep 17 01:44:37.380718 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.380700    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-openshift-service-ca\") pod \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\" (UID: \"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154\") "
Sep 17 01:44:37.381208 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.381175    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d92afd7-06f7-4ad4-b8b2-9233fcfd3154" (UID: "6d92afd7-06f7-4ad4-b8b2-9233fcfd3154"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:44:37.382819 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.382780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d92afd7-06f7-4ad4-b8b2-9233fcfd3154" (UID: "6d92afd7-06f7-4ad4-b8b2-9233fcfd3154"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:44:37.382913 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.382847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d92afd7-06f7-4ad4-b8b2-9233fcfd3154" (UID: "6d92afd7-06f7-4ad4-b8b2-9233fcfd3154"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:44:37.382913 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.382886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-kube-api-access-vr65m" (OuterVolumeSpecName: "kube-api-access-vr65m") pod "6d92afd7-06f7-4ad4-b8b2-9233fcfd3154" (UID: "6d92afd7-06f7-4ad4-b8b2-9233fcfd3154"). InnerVolumeSpecName "kube-api-access-vr65m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:44:37.482376 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.482340    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-maas-api-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:37.482376 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.482370    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr65m\" (UniqueName: \"kubernetes.io/projected/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-kube-api-access-vr65m\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:37.482376 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.482381    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-metrics-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:37.482627 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.482389    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154-openshift-service-ca\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:37.740586 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.740474    2578 generic.go:356] "Generic (PLEG): container finished" podID="6d92afd7-06f7-4ad4-b8b2-9233fcfd3154" containerID="4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb" exitCode=0
Sep 17 01:44:37.740586 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.740522    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"
Sep 17 01:44:37.740586 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.740560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486" event={"ID":"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154","Type":"ContainerDied","Data":"4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb"}
Sep 17 01:44:37.740871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.740615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486" event={"ID":"6d92afd7-06f7-4ad4-b8b2-9233fcfd3154","Type":"ContainerDied","Data":"3e12cc5820e31fc3d7193b917ee45c530e5311c3ac7e27e48763754e5c43cd4b"}
Sep 17 01:44:37.740871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.740639    2578 scope.go:117] "RemoveContainer" containerID="4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb"
Sep 17 01:44:37.750343 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.750326    2578 scope.go:117] "RemoveContainer" containerID="4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb"
Sep 17 01:44:37.750607 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:44:37.750590    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb\": container with ID starting with 4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb not found: ID does not exist" containerID="4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb"
Sep 17 01:44:37.750673 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.750615    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb"} err="failed to get container status \"4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb\": rpc error: code = NotFound desc = could not find container \"4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb\": container with ID starting with 4d3965350f8234261dbbd8be472913fd8b63fd15c4d2c4ac97d1020360056bdb not found: ID does not exist"
Sep 17 01:44:37.760969 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.760940    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"]
Sep 17 01:44:37.765273 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.765237    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d8d689-cbbcb48f8-4w486"]
Sep 17 01:44:37.920812 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:37.920781    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d92afd7-06f7-4ad4-b8b2-9233fcfd3154" path="/var/lib/kubelet/pods/6d92afd7-06f7-4ad4-b8b2-9233fcfd3154/volumes"
Sep 17 01:44:46.162216 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:46.162181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:46.162688 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:46.162242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:46.174883 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:46.174861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:46.796387 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:46.796357    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:44:55.513758 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.513722    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"]
Sep 17 01:44:55.514261 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.514008    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67" podUID="9ae8a13e-21f0-46d5-964b-6b164df2a269" containerName="maas-api" containerID="cri-o://b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28" gracePeriod=30
Sep 17 01:44:55.760618 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.760594    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:44:55.830209 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.830178    2578 generic.go:356] "Generic (PLEG): container finished" podID="9ae8a13e-21f0-46d5-964b-6b164df2a269" containerID="b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28" exitCode=0
Sep 17 01:44:55.830358 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.830230    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"
Sep 17 01:44:55.830358 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.830266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67" event={"ID":"9ae8a13e-21f0-46d5-964b-6b164df2a269","Type":"ContainerDied","Data":"b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28"}
Sep 17 01:44:55.830358 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.830317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67" event={"ID":"9ae8a13e-21f0-46d5-964b-6b164df2a269","Type":"ContainerDied","Data":"36e786545c130c870bf7023fe6d45152c000477b21e0d4ab0ac654a086e69b09"}
Sep 17 01:44:55.830358 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.830340    2578 scope.go:117] "RemoveContainer" containerID="b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28"
Sep 17 01:44:55.839277 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.839255    2578 scope.go:117] "RemoveContainer" containerID="b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28"
Sep 17 01:44:55.839584 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:44:55.839562    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28\": container with ID starting with b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28 not found: ID does not exist" containerID="b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28"
Sep 17 01:44:55.839658 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.839591    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28"} err="failed to get container status \"b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28\": rpc error: code = NotFound desc = could not find container \"b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28\": container with ID starting with b2658486a918d524885e9c06857586844ee2035d12a699eda48163bd8a7cdd28 not found: ID does not exist"
Sep 17 01:44:55.883899 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.883870    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-maas-api-tls\") pod \"9ae8a13e-21f0-46d5-964b-6b164df2a269\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") "
Sep 17 01:44:55.884069 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.883924    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae8a13e-21f0-46d5-964b-6b164df2a269-openshift-service-ca\") pod \"9ae8a13e-21f0-46d5-964b-6b164df2a269\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") "
Sep 17 01:44:55.884069 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.883950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9dvk\" (UniqueName: \"kubernetes.io/projected/9ae8a13e-21f0-46d5-964b-6b164df2a269-kube-api-access-b9dvk\") pod \"9ae8a13e-21f0-46d5-964b-6b164df2a269\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") "
Sep 17 01:44:55.884069 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.883997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-metrics-certs\") pod \"9ae8a13e-21f0-46d5-964b-6b164df2a269\" (UID: \"9ae8a13e-21f0-46d5-964b-6b164df2a269\") "
Sep 17 01:44:55.884272 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.884246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae8a13e-21f0-46d5-964b-6b164df2a269-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ae8a13e-21f0-46d5-964b-6b164df2a269" (UID: "9ae8a13e-21f0-46d5-964b-6b164df2a269"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:44:55.884426 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.884410    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae8a13e-21f0-46d5-964b-6b164df2a269-openshift-service-ca\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:55.886085 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.886063    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae8a13e-21f0-46d5-964b-6b164df2a269-kube-api-access-b9dvk" (OuterVolumeSpecName: "kube-api-access-b9dvk") pod "9ae8a13e-21f0-46d5-964b-6b164df2a269" (UID: "9ae8a13e-21f0-46d5-964b-6b164df2a269"). InnerVolumeSpecName "kube-api-access-b9dvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:44:55.886517 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.886491    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ae8a13e-21f0-46d5-964b-6b164df2a269" (UID: "9ae8a13e-21f0-46d5-964b-6b164df2a269"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:44:55.886517 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.886505    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ae8a13e-21f0-46d5-964b-6b164df2a269" (UID: "9ae8a13e-21f0-46d5-964b-6b164df2a269"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:44:55.985580 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.985527    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-maas-api-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:55.985580 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.985577    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9dvk\" (UniqueName: \"kubernetes.io/projected/9ae8a13e-21f0-46d5-964b-6b164df2a269-kube-api-access-b9dvk\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:55.985580 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:55.985591    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ae8a13e-21f0-46d5-964b-6b164df2a269-metrics-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:44:56.147611 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:56.147520    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"]
Sep 17 01:44:56.149372 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:56.149348    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d370b3-5bf7c69c99-v7x67"]
Sep 17 01:44:57.921053 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:44:57.921009    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae8a13e-21f0-46d5-964b-6b164df2a269" path="/var/lib/kubelet/pods/9ae8a13e-21f0-46d5-964b-6b164df2a269/volumes"
Sep 17 01:45:26.076587 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.076554    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"]
Sep 17 01:45:26.077330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.077135    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d92afd7-06f7-4ad4-b8b2-9233fcfd3154" containerName="maas-api"
Sep 17 01:45:26.077330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.077156    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d92afd7-06f7-4ad4-b8b2-9233fcfd3154" containerName="maas-api"
Sep 17 01:45:26.077330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.077167    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae8a13e-21f0-46d5-964b-6b164df2a269" containerName="maas-api"
Sep 17 01:45:26.077330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.077175    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae8a13e-21f0-46d5-964b-6b164df2a269" containerName="maas-api"
Sep 17 01:45:26.077330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.077284    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d92afd7-06f7-4ad4-b8b2-9233fcfd3154" containerName="maas-api"
Sep 17 01:45:26.077330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.077298    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae8a13e-21f0-46d5-964b-6b164df2a269" containerName="maas-api"
Sep 17 01:45:26.085869 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.085843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.088342 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.088311    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"]
Sep 17 01:45:26.089512 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.089489    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2a1db100-serving-cert\""
Sep 17 01:45:26.188582 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.188530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dgpz\" (UniqueName: \"kubernetes.io/projected/933c6748-c325-4639-b905-776f3102b63d-kube-api-access-6dgpz\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.188771 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.188609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-maas-api-tls\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.188771 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.188715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-metrics-certs\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.188887 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.188814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933c6748-c325-4639-b905-776f3102b63d-openshift-service-ca\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.289786 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.289748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-metrics-certs\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.289997 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.289844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933c6748-c325-4639-b905-776f3102b63d-openshift-service-ca\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.289997 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.289877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dgpz\" (UniqueName: \"kubernetes.io/projected/933c6748-c325-4639-b905-776f3102b63d-kube-api-access-6dgpz\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.289997 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.289908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-maas-api-tls\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.290585 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.290552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933c6748-c325-4639-b905-776f3102b63d-openshift-service-ca\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.292322 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.292300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-maas-api-tls\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.292427 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.292362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-metrics-certs\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.301879 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.301845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dgpz\" (UniqueName: \"kubernetes.io/projected/933c6748-c325-4639-b905-776f3102b63d-kube-api-access-6dgpz\") pod \"maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.400142 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.400001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:26.674790 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.674761    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"]
Sep 17 01:45:26.676952 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:45:26.676922    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933c6748_c325_4639_b905_776f3102b63d.slice/crio-c28b1e3f5a832a0b88dd8247829b2a9d7326ecc2975f291145aec87d6c956e4e WatchSource:0}: Error finding container c28b1e3f5a832a0b88dd8247829b2a9d7326ecc2975f291145aec87d6c956e4e: Status 404 returned error can't find the container with id c28b1e3f5a832a0b88dd8247829b2a9d7326ecc2975f291145aec87d6c956e4e
Sep 17 01:45:26.678283 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.678265    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 01:45:26.922716 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.922683    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"]
Sep 17 01:45:26.927604 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.927519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:45:26.930027 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.930005    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2a1db100\""
Sep 17 01:45:26.930199 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.930186    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2a1db100-dockercfg-trb9p\""
Sep 17 01:45:26.936332 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.936306    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"]
Sep 17 01:45:26.997902 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.997864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6\" (UID: \"cdbc353f-9ddb-46e3-92ba-5379c9a826dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:45:26.998088 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.997945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7vm2\" (UniqueName: \"kubernetes.io/projected/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-kube-api-access-s7vm2\") pod \"payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6\" (UID: \"cdbc353f-9ddb-46e3-92ba-5379c9a826dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:45:26.998619 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:26.998592    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf" event={"ID":"933c6748-c325-4639-b905-776f3102b63d","Type":"ContainerStarted","Data":"c28b1e3f5a832a0b88dd8247829b2a9d7326ecc2975f291145aec87d6c956e4e"}
Sep 17 01:45:27.099120 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:27.099083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6\" (UID: \"cdbc353f-9ddb-46e3-92ba-5379c9a826dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:45:27.099515 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:27.099129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7vm2\" (UniqueName: \"kubernetes.io/projected/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-kube-api-access-s7vm2\") pod \"payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6\" (UID: \"cdbc353f-9ddb-46e3-92ba-5379c9a826dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:45:27.099845 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:27.099825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6\" (UID: \"cdbc353f-9ddb-46e3-92ba-5379c9a826dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:45:27.106667 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:27.106641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7vm2\" (UniqueName: \"kubernetes.io/projected/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-kube-api-access-s7vm2\") pod \"payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6\" (UID: \"cdbc353f-9ddb-46e3-92ba-5379c9a826dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:45:27.240642 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:27.240617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:45:27.516919 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:27.516896    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"]
Sep 17 01:45:27.519180 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:45:27.519153    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbc353f_9ddb_46e3_92ba_5379c9a826dd.slice/crio-08a9c98717f3f66e617f91dc75066b21399f490741f4936da7a52b1d295db900 WatchSource:0}: Error finding container 08a9c98717f3f66e617f91dc75066b21399f490741f4936da7a52b1d295db900: Status 404 returned error can't find the container with id 08a9c98717f3f66e617f91dc75066b21399f490741f4936da7a52b1d295db900
Sep 17 01:45:28.004463 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:28.004428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6" event={"ID":"cdbc353f-9ddb-46e3-92ba-5379c9a826dd","Type":"ContainerStarted","Data":"08a9c98717f3f66e617f91dc75066b21399f490741f4936da7a52b1d295db900"}
Sep 17 01:45:28.005971 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:28.005946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf" event={"ID":"933c6748-c325-4639-b905-776f3102b63d","Type":"ContainerStarted","Data":"cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4"}
Sep 17 01:45:28.006124 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:28.006034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:28.028847 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:28.028790    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf" podStartSLOduration=1.4934557339999999 podStartE2EDuration="2.028768957s" totalImagesPullingTime="535.313223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:45:28.02514489 +0000 UTC m=+1406.695660980" watchObservedRunningTime="2026-09-17 01:45:28.028768957 +0000 UTC m=+1406.699285047"
Sep 17 01:45:30.020061 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:30.019938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6" event={"ID":"cdbc353f-9ddb-46e3-92ba-5379c9a826dd","Type":"ContainerStarted","Data":"3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13"}
Sep 17 01:45:30.020576 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:30.020223    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:45:30.038043 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:30.037983    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6" podStartSLOduration=1.80063722 podStartE2EDuration="4.037958917s" totalImagesPullingTime="2.237321697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:45:30.035093946 +0000 UTC m=+1408.705610040" watchObservedRunningTime="2026-09-17 01:45:30.037958917 +0000 UTC m=+1408.708475008"
Sep 17 01:45:34.017516 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:34.017477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:45:41.027784 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:45:41.027745    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:46:16.750747 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.750712    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"]
Sep 17 01:46:16.754733 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.754705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.757259 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.757227    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2a1db100\"/\"openshift-service-ca.crt\""
Sep 17 01:46:16.757259 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.757252    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2a1db100\"/\"e2e-lifecycle-model-2a1db100-kserve-self-signed-certs\""
Sep 17 01:46:16.758629 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.758570    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2a1db100\"/\"default-dockercfg-vhvmz\""
Sep 17 01:46:16.758803 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.758605    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2a1db100\"/\"kube-root-ca.crt\""
Sep 17 01:46:16.761469 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.761446    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"]
Sep 17 01:46:16.800845 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.800814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-home\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.801027 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.800861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tls-certs\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.801027 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.800882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85stv\" (UniqueName: \"kubernetes.io/projected/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-kube-api-access-85stv\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.801027 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.800957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-model-cache\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.801027 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.801010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-dshm\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.801168 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.801131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tmp-dir\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.902622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.902565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tmp-dir\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.902787 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.902660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-home\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.902787 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.902702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tls-certs\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.902787 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.902719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85stv\" (UniqueName: \"kubernetes.io/projected/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-kube-api-access-85stv\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.902787 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.902758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-model-cache\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.902942 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.902815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-dshm\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.902942 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.902920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tmp-dir\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.903045 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.903023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-home\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.903084 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.903050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-model-cache\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.905093 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.905064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-dshm\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.905235 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.905213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tls-certs\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:16.910041 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:16.910017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85stv\" (UniqueName: \"kubernetes.io/projected/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-kube-api-access-85stv\") pod \"e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") " pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:17.067199 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:17.067159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:17.340842 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:17.339256    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"]
Sep 17 01:46:18.257123 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:18.257085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw" event={"ID":"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4","Type":"ContainerStarted","Data":"dd5d3758046bbdfbdde8e948bd7d3870123e09bd723c9bbb3b806963d9a602b6"}
Sep 17 01:46:18.257529 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:18.257132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw" event={"ID":"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4","Type":"ContainerStarted","Data":"2cfcf0e2a2bae1480ba569bb1463037f9dad23090d5dc10f9d500f7e1858c3ca"}
Sep 17 01:46:18.275250 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:18.275189    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw" podStartSLOduration=2.275173927 podStartE2EDuration="2.275173927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:46:18.273182147 +0000 UTC m=+1456.943698437" watchObservedRunningTime="2026-09-17 01:46:18.275173927 +0000 UTC m=+1456.945690042"
Sep 17 01:46:27.067596 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:27.067559    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:27.067596 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:27.067605    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:27.080597 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:27.080569    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:27.314611 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:27.314583    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:46:43.463420 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.463379    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"]
Sep 17 01:46:43.467102 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.467082    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.470413 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.470381    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-fcd402\"/\"kube-root-ca.crt\""
Sep 17 01:46:43.470563 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.470491    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-fcd402\"/\"auth-test-model-fcd402-kserve-self-signed-certs\""
Sep 17 01:46:43.470644 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.470582    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-fcd402\"/\"default-dockercfg-gxc5s\""
Sep 17 01:46:43.470705 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.470644    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-fcd402\"/\"openshift-service-ca.crt\""
Sep 17 01:46:43.476233 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.476209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"]
Sep 17 01:46:43.573107 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.573071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-model-cache\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.573267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.573126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-home\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.573267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.573143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f93183-62ab-42e8-9cdc-a72a86515cee-tls-certs\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.573267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.573205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-dshm\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.573376 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.573267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-tmp-dir\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.573376 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.573329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8njj7\" (UniqueName: \"kubernetes.io/projected/43f93183-62ab-42e8-9cdc-a72a86515cee-kube-api-access-8njj7\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.674936 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.674908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-dshm\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.675116 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.674971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-tmp-dir\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.675116 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.674998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8njj7\" (UniqueName: \"kubernetes.io/projected/43f93183-62ab-42e8-9cdc-a72a86515cee-kube-api-access-8njj7\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.675116 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.675037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-model-cache\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.675116 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.675105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-home\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.675349 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.675143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f93183-62ab-42e8-9cdc-a72a86515cee-tls-certs\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.675482 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.675452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-tmp-dir\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.675579 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.675477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-model-cache\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.675579 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.675497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-home\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.677289 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.677263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-dshm\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.677519 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.677504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f93183-62ab-42e8-9cdc-a72a86515cee-tls-certs\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.682033 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.682014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8njj7\" (UniqueName: \"kubernetes.io/projected/43f93183-62ab-42e8-9cdc-a72a86515cee-kube-api-access-8njj7\") pod \"auth-test-model-fcd402-kserve-5fcc65798c-9knl8\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:43.779049 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:43.778951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:44.052787 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:44.052755    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"]
Sep 17 01:46:44.054573 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:46:44.054520    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43f93183_62ab_42e8_9cdc_a72a86515cee.slice/crio-b582471f0111efdc2e06f2a330792b4623d47753e7590e03a0c4c5b06eaf962f WatchSource:0}: Error finding container b582471f0111efdc2e06f2a330792b4623d47753e7590e03a0c4c5b06eaf962f: Status 404 returned error can't find the container with id b582471f0111efdc2e06f2a330792b4623d47753e7590e03a0c4c5b06eaf962f
Sep 17 01:46:44.397481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:44.397398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8" event={"ID":"43f93183-62ab-42e8-9cdc-a72a86515cee","Type":"ContainerStarted","Data":"55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7"}
Sep 17 01:46:44.397481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:44.397438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8" event={"ID":"43f93183-62ab-42e8-9cdc-a72a86515cee","Type":"ContainerStarted","Data":"b582471f0111efdc2e06f2a330792b4623d47753e7590e03a0c4c5b06eaf962f"}
Sep 17 01:46:44.412984 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:44.412935    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8" podStartSLOduration=1.4129199940000001 podStartE2EDuration="1.412919994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:46:44.411460863 +0000 UTC m=+1483.081976951" watchObservedRunningTime="2026-09-17 01:46:44.412919994 +0000 UTC m=+1483.083436099"
Sep 17 01:46:53.779191 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:53.779143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:53.779191 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:53.779205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:53.792016 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:53.791992    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:46:54.463797 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:46:54.463768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:47:11.699564 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:11.699514    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"]
Sep 17 01:47:11.703836 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:11.703804    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"]
Sep 17 01:47:11.727497 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:11.727468    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"]
Sep 17 01:47:22.396839 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.396808    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"]
Sep 17 01:47:22.397402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.397031    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np" podUID="521b99ff-be70-4e0c-8464-e560b8e2fe46" containerName="maas-api" containerID="cri-o://b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f" gracePeriod=30
Sep 17 01:47:22.858282 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.858255    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:47:22.995235 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.995145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-maas-api-tls\") pod \"521b99ff-be70-4e0c-8464-e560b8e2fe46\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") "
Sep 17 01:47:22.995235 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.995203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqvvr\" (UniqueName: \"kubernetes.io/projected/521b99ff-be70-4e0c-8464-e560b8e2fe46-kube-api-access-qqvvr\") pod \"521b99ff-be70-4e0c-8464-e560b8e2fe46\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") "
Sep 17 01:47:22.995477 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.995265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-metrics-certs\") pod \"521b99ff-be70-4e0c-8464-e560b8e2fe46\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") "
Sep 17 01:47:22.995477 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.995283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521b99ff-be70-4e0c-8464-e560b8e2fe46-openshift-service-ca\") pod \"521b99ff-be70-4e0c-8464-e560b8e2fe46\" (UID: \"521b99ff-be70-4e0c-8464-e560b8e2fe46\") "
Sep 17 01:47:22.995729 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.995703    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521b99ff-be70-4e0c-8464-e560b8e2fe46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "521b99ff-be70-4e0c-8464-e560b8e2fe46" (UID: "521b99ff-be70-4e0c-8464-e560b8e2fe46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:47:22.997403 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.997373    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521b99ff-be70-4e0c-8464-e560b8e2fe46-kube-api-access-qqvvr" (OuterVolumeSpecName: "kube-api-access-qqvvr") pod "521b99ff-be70-4e0c-8464-e560b8e2fe46" (UID: "521b99ff-be70-4e0c-8464-e560b8e2fe46"). InnerVolumeSpecName "kube-api-access-qqvvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:47:22.997512 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.997373    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "521b99ff-be70-4e0c-8464-e560b8e2fe46" (UID: "521b99ff-be70-4e0c-8464-e560b8e2fe46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:47:22.997512 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:22.997447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "521b99ff-be70-4e0c-8464-e560b8e2fe46" (UID: "521b99ff-be70-4e0c-8464-e560b8e2fe46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:47:23.096390 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.096349    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-maas-api-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:23.096390 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.096386    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqvvr\" (UniqueName: \"kubernetes.io/projected/521b99ff-be70-4e0c-8464-e560b8e2fe46-kube-api-access-qqvvr\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:23.096390 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.096397    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521b99ff-be70-4e0c-8464-e560b8e2fe46-metrics-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:23.096656 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.096409    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521b99ff-be70-4e0c-8464-e560b8e2fe46-openshift-service-ca\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:23.613495 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.613458    2578 generic.go:356] "Generic (PLEG): container finished" podID="521b99ff-be70-4e0c-8464-e560b8e2fe46" containerID="b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f" exitCode=0
Sep 17 01:47:23.613910 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.613507    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"
Sep 17 01:47:23.613910 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.613565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np" event={"ID":"521b99ff-be70-4e0c-8464-e560b8e2fe46","Type":"ContainerDied","Data":"b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f"}
Sep 17 01:47:23.613910 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.613619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np" event={"ID":"521b99ff-be70-4e0c-8464-e560b8e2fe46","Type":"ContainerDied","Data":"70da8896260c40d9b7737916ea8cfac33076bbde9145477708f4bea0ee477a62"}
Sep 17 01:47:23.613910 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.613641    2578 scope.go:117] "RemoveContainer" containerID="b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f"
Sep 17 01:47:23.624755 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.624733    2578 scope.go:117] "RemoveContainer" containerID="b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f"
Sep 17 01:47:23.625013 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:47:23.624994    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f\": container with ID starting with b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f not found: ID does not exist" containerID="b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f"
Sep 17 01:47:23.625059 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.625024    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f"} err="failed to get container status \"b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f\": rpc error: code = NotFound desc = could not find container \"b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f\": container with ID starting with b86166c9a117585634abf6cd48da10a2e61ef1f66ffb075e000a97413699cc0f not found: ID does not exist"
Sep 17 01:47:23.637377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.637353    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"]
Sep 17 01:47:23.641157 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.641129    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cc98e0-695d56c4b5-2d2np"]
Sep 17 01:47:23.920900 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:23.920795    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="521b99ff-be70-4e0c-8464-e560b8e2fe46" path="/var/lib/kubelet/pods/521b99ff-be70-4e0c-8464-e560b8e2fe46/volumes"
Sep 17 01:47:26.719559 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:26.719499    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj" podUID="9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" containerName="main" containerID="cri-o://4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c" gracePeriod=45
Sep 17 01:47:26.723257 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:26.723220    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf" podUID="c6f37728-182c-44b2-a837-8f772236f67b" containerName="main" containerID="cri-o://4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4" gracePeriod=45
Sep 17 01:47:26.750001 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:26.749969    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78" podUID="14053846-72b9-4e45-987d-a26675e0ebb8" containerName="main" containerID="cri-o://78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6" gracePeriod=45
Sep 17 01:47:27.022467 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.022438    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:47:27.036970 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.036948    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:47:27.044557 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.044512    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:47:27.143826 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.143794    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-tmp-dir\") pod \"14053846-72b9-4e45-987d-a26675e0ebb8\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") "
Sep 17 01:47:27.143974 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.143857    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-dshm\") pod \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") "
Sep 17 01:47:27.143974 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.143877    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-dshm\") pod \"c6f37728-182c-44b2-a837-8f772236f67b\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") "
Sep 17 01:47:27.143974 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.143902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-tmp-dir\") pod \"c6f37728-182c-44b2-a837-8f772236f67b\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") "
Sep 17 01:47:27.143974 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.143943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6mkw\" (UniqueName: \"kubernetes.io/projected/c6f37728-182c-44b2-a837-8f772236f67b-kube-api-access-c6mkw\") pod \"c6f37728-182c-44b2-a837-8f772236f67b\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") "
Sep 17 01:47:27.143974 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.143961    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-home\") pod \"c6f37728-182c-44b2-a837-8f772236f67b\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") "
Sep 17 01:47:27.144280 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb4zt\" (UniqueName: \"kubernetes.io/projected/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-kube-api-access-sb4zt\") pod \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") "
Sep 17 01:47:27.144280 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-home\") pod \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") "
Sep 17 01:47:27.144280 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144078    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14053846-72b9-4e45-987d-a26675e0ebb8" (UID: "14053846-72b9-4e45-987d-a26675e0ebb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.144280 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f37728-182c-44b2-a837-8f772236f67b-tls-certs\") pod \"c6f37728-182c-44b2-a837-8f772236f67b\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") "
Sep 17 01:47:27.144280 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144188    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tmp-dir\") pod \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") "
Sep 17 01:47:27.144280 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14053846-72b9-4e45-987d-a26675e0ebb8-tls-certs\") pod \"14053846-72b9-4e45-987d-a26675e0ebb8\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") "
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144279    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-home" (OuterVolumeSpecName: "home") pod "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" (UID: "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-home" (OuterVolumeSpecName: "home") pod "c6f37728-182c-44b2-a837-8f772236f67b" (UID: "c6f37728-182c-44b2-a837-8f772236f67b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-model-cache\") pod \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") "
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-model-cache\") pod \"c6f37728-182c-44b2-a837-8f772236f67b\" (UID: \"c6f37728-182c-44b2-a837-8f772236f67b\") "
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144390    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tls-certs\") pod \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\" (UID: \"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7\") "
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" (UID: "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144454    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-home\") pod \"14053846-72b9-4e45-987d-a26675e0ebb8\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") "
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" (UID: "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6f37728-182c-44b2-a837-8f772236f67b" (UID: "c6f37728-182c-44b2-a837-8f772236f67b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-model-cache\") pod \"14053846-72b9-4e45-987d-a26675e0ebb8\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") "
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-dshm\") pod \"14053846-72b9-4e45-987d-a26675e0ebb8\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") "
Sep 17 01:47:27.144622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8kbm\" (UniqueName: \"kubernetes.io/projected/14053846-72b9-4e45-987d-a26675e0ebb8-kube-api-access-r8kbm\") pod \"14053846-72b9-4e45-987d-a26675e0ebb8\" (UID: \"14053846-72b9-4e45-987d-a26675e0ebb8\") "
Sep 17 01:47:27.145252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144635    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-home" (OuterVolumeSpecName: "home") pod "14053846-72b9-4e45-987d-a26675e0ebb8" (UID: "14053846-72b9-4e45-987d-a26675e0ebb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.145252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.144827    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6f37728-182c-44b2-a837-8f772236f67b" (UID: "c6f37728-182c-44b2-a837-8f772236f67b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.145252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.145033    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.145252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.145052    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.145252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.145064    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.145252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.145075    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.145252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.145087    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.145252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.145099    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.145252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.145111    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.145252 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.145123    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.145784 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.145277    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "14053846-72b9-4e45-987d-a26675e0ebb8" (UID: "14053846-72b9-4e45-987d-a26675e0ebb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.146822 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.146780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-dshm" (OuterVolumeSpecName: "dshm") pod "c6f37728-182c-44b2-a837-8f772236f67b" (UID: "c6f37728-182c-44b2-a837-8f772236f67b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.146977 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.146819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-dshm" (OuterVolumeSpecName: "dshm") pod "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" (UID: "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.146977 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.146833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f37728-182c-44b2-a837-8f772236f67b-kube-api-access-c6mkw" (OuterVolumeSpecName: "kube-api-access-c6mkw") pod "c6f37728-182c-44b2-a837-8f772236f67b" (UID: "c6f37728-182c-44b2-a837-8f772236f67b"). InnerVolumeSpecName "kube-api-access-c6mkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:47:27.147103 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.147005    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f37728-182c-44b2-a837-8f772236f67b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6f37728-182c-44b2-a837-8f772236f67b" (UID: "c6f37728-182c-44b2-a837-8f772236f67b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:47:27.147380 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.147359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" (UID: "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:47:27.147528 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.147507    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-kube-api-access-sb4zt" (OuterVolumeSpecName: "kube-api-access-sb4zt") pod "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" (UID: "9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7"). InnerVolumeSpecName "kube-api-access-sb4zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:47:27.147680 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.147662    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14053846-72b9-4e45-987d-a26675e0ebb8-kube-api-access-r8kbm" (OuterVolumeSpecName: "kube-api-access-r8kbm") pod "14053846-72b9-4e45-987d-a26675e0ebb8" (UID: "14053846-72b9-4e45-987d-a26675e0ebb8"). InnerVolumeSpecName "kube-api-access-r8kbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:47:27.148075 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.148055    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14053846-72b9-4e45-987d-a26675e0ebb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14053846-72b9-4e45-987d-a26675e0ebb8" (UID: "14053846-72b9-4e45-987d-a26675e0ebb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:47:27.148156 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.148133    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-dshm" (OuterVolumeSpecName: "dshm") pod "14053846-72b9-4e45-987d-a26675e0ebb8" (UID: "14053846-72b9-4e45-987d-a26675e0ebb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:47:27.246583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.246485    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.246583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.246511    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14053846-72b9-4e45-987d-a26675e0ebb8-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.246583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.246521    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8kbm\" (UniqueName: \"kubernetes.io/projected/14053846-72b9-4e45-987d-a26675e0ebb8-kube-api-access-r8kbm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.246583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.246530    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.246583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.246568    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f37728-182c-44b2-a837-8f772236f67b-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.246583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.246576    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6mkw\" (UniqueName: \"kubernetes.io/projected/c6f37728-182c-44b2-a837-8f772236f67b-kube-api-access-c6mkw\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.246583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.246584    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb4zt\" (UniqueName: \"kubernetes.io/projected/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-kube-api-access-sb4zt\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.246904 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.246593    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f37728-182c-44b2-a837-8f772236f67b-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.246904 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.246601    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14053846-72b9-4e45-987d-a26675e0ebb8-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.246904 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.246608    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:47:27.634447 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.634415    2578 generic.go:356] "Generic (PLEG): container finished" podID="14053846-72b9-4e45-987d-a26675e0ebb8" containerID="78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6" exitCode=0
Sep 17 01:47:27.634651 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.634464    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"
Sep 17 01:47:27.634651 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.634503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78" event={"ID":"14053846-72b9-4e45-987d-a26675e0ebb8","Type":"ContainerDied","Data":"78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6"}
Sep 17 01:47:27.634651 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.634577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78" event={"ID":"14053846-72b9-4e45-987d-a26675e0ebb8","Type":"ContainerDied","Data":"11bc07e8ec4abcdfae0880a823556ce042e45cc90f91541bb62c24b2a91e09ce"}
Sep 17 01:47:27.634651 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.634603    2578 scope.go:117] "RemoveContainer" containerID="78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6"
Sep 17 01:47:27.636016 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.635985    2578 generic.go:356] "Generic (PLEG): container finished" podID="9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" containerID="4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c" exitCode=0
Sep 17 01:47:27.636121 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.636021    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"
Sep 17 01:47:27.636121 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.636076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj" event={"ID":"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7","Type":"ContainerDied","Data":"4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c"}
Sep 17 01:47:27.636121 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.636116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj" event={"ID":"9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7","Type":"ContainerDied","Data":"fd3009874c303a87fdf8f392d1e6965c8f39c76de6d958c5d834e5578dbce55b"}
Sep 17 01:47:27.637710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.637678    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6f37728-182c-44b2-a837-8f772236f67b" containerID="4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4" exitCode=0
Sep 17 01:47:27.637819 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.637748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf" event={"ID":"c6f37728-182c-44b2-a837-8f772236f67b","Type":"ContainerDied","Data":"4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4"}
Sep 17 01:47:27.637819 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.637770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf" event={"ID":"c6f37728-182c-44b2-a837-8f772236f67b","Type":"ContainerDied","Data":"903c1be40b27b6dad659ba4ab29c0d59796176f562f46c1d16f846d7c64da998"}
Sep 17 01:47:27.637819 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.637792    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"
Sep 17 01:47:27.645438 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.645222    2578 scope.go:117] "RemoveContainer" containerID="78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6"
Sep 17 01:47:27.645587 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:47:27.645563    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6\": container with ID starting with 78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6 not found: ID does not exist" containerID="78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6"
Sep 17 01:47:27.645679 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.645597    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6"} err="failed to get container status \"78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6\": rpc error: code = NotFound desc = could not find container \"78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6\": container with ID starting with 78525ad945e74cc507faf2f112c5045adddd77e168808bf062156f0b5fa351e6 not found: ID does not exist"
Sep 17 01:47:27.645679 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.645613    2578 scope.go:117] "RemoveContainer" containerID="4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c"
Sep 17 01:47:27.653390 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.653367    2578 scope.go:117] "RemoveContainer" containerID="4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c"
Sep 17 01:47:27.653679 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:47:27.653662    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c\": container with ID starting with 4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c not found: ID does not exist" containerID="4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c"
Sep 17 01:47:27.653755 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.653683    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c"} err="failed to get container status \"4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c\": rpc error: code = NotFound desc = could not find container \"4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c\": container with ID starting with 4a6f5320190e47880f012e336a4c3da3a04f5f720a4f5abe0ef8bd7633c3614c not found: ID does not exist"
Sep 17 01:47:27.653755 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.653697    2578 scope.go:117] "RemoveContainer" containerID="4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4"
Sep 17 01:47:27.661968 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.661949    2578 scope.go:117] "RemoveContainer" containerID="4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4"
Sep 17 01:47:27.662291 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:47:27.662269    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4\": container with ID starting with 4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4 not found: ID does not exist" containerID="4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4"
Sep 17 01:47:27.662360 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.662299    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4"} err="failed to get container status \"4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4\": rpc error: code = NotFound desc = could not find container \"4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4\": container with ID starting with 4ce2f83d6380a938b2f694d27d0cc7cef2dceb25cdc24693374bba6b01230ad4 not found: ID does not exist"
Sep 17 01:47:27.664725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.664702    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"]
Sep 17 01:47:27.669030 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.669009    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-2-w3-cc98e0-kserve-655cc7965c-l7fxj"]
Sep 17 01:47:27.684061 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.684030    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"]
Sep 17 01:47:27.691323 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.691302    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/simulator-w3-cc98e0-kserve-68756b55c4-6kf78"]
Sep 17 01:47:27.702842 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.702819    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"]
Sep 17 01:47:27.705800 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.705779    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/distinct-w3-cc98e0-kserve-7dfcd7f8b-2mzkf"]
Sep 17 01:47:27.921514 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.921423    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14053846-72b9-4e45-987d-a26675e0ebb8" path="/var/lib/kubelet/pods/14053846-72b9-4e45-987d-a26675e0ebb8/volumes"
Sep 17 01:47:27.922072 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.922050    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" path="/var/lib/kubelet/pods/9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7/volumes"
Sep 17 01:47:27.922484 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:27.922466    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f37728-182c-44b2-a837-8f772236f67b" path="/var/lib/kubelet/pods/c6f37728-182c-44b2-a837-8f772236f67b/volumes"
Sep 17 01:47:36.657091 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657049    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"]
Sep 17 01:47:36.657573 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657549    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14053846-72b9-4e45-987d-a26675e0ebb8" containerName="main"
Sep 17 01:47:36.657687 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657575    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="14053846-72b9-4e45-987d-a26675e0ebb8" containerName="main"
Sep 17 01:47:36.657687 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657590    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f37728-182c-44b2-a837-8f772236f67b" containerName="main"
Sep 17 01:47:36.657687 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657598    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f37728-182c-44b2-a837-8f772236f67b" containerName="main"
Sep 17 01:47:36.657687 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657609    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" containerName="main"
Sep 17 01:47:36.657687 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657617    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" containerName="main"
Sep 17 01:47:36.657687 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657632    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="521b99ff-be70-4e0c-8464-e560b8e2fe46" containerName="maas-api"
Sep 17 01:47:36.657687 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657641    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="521b99ff-be70-4e0c-8464-e560b8e2fe46" containerName="maas-api"
Sep 17 01:47:36.657926 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657754    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="521b99ff-be70-4e0c-8464-e560b8e2fe46" containerName="maas-api"
Sep 17 01:47:36.657926 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657765    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f37728-182c-44b2-a837-8f772236f67b" containerName="main"
Sep 17 01:47:36.657926 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657771    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="14053846-72b9-4e45-987d-a26675e0ebb8" containerName="main"
Sep 17 01:47:36.657926 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.657778    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9074d984-88b4-4b9f-a2ce-eb9aeb50a4b7" containerName="main"
Sep 17 01:47:36.661092 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.661072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.670367 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.670337    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-361849\"/\"default-dockercfg-vqzvr\""
Sep 17 01:47:36.673459 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.671177    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-361849\"/\"kube-root-ca.crt\""
Sep 17 01:47:36.673459 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.671620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-361849\"/\"auth-test-model-361849-kserve-self-signed-certs\""
Sep 17 01:47:36.673459 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.672006    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-361849\"/\"openshift-service-ca.crt\""
Sep 17 01:47:36.677283 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.677259    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"]
Sep 17 01:47:36.745419 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.745380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-dshm\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.745600 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.745435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c39857-81ae-4d39-a06d-95f4e115951b-tls-certs\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.745600 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.745513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28vcs\" (UniqueName: \"kubernetes.io/projected/f3c39857-81ae-4d39-a06d-95f4e115951b-kube-api-access-28vcs\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.745600 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.745578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-tmp-dir\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.745710 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.745658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-home\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.745746 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.745737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-model-cache\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.847004 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.846976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28vcs\" (UniqueName: \"kubernetes.io/projected/f3c39857-81ae-4d39-a06d-95f4e115951b-kube-api-access-28vcs\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.847192 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.847015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-tmp-dir\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.847192 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.847039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-home\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.847192 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.847097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-model-cache\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.847192 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.847155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-dshm\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.847379 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.847202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c39857-81ae-4d39-a06d-95f4e115951b-tls-certs\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.847413 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.847394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-tmp-dir\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.847490 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.847447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-home\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.847490 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.847468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-model-cache\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.849571 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.849547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-dshm\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.849689 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.849675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c39857-81ae-4d39-a06d-95f4e115951b-tls-certs\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.854478 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.854453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28vcs\" (UniqueName: \"kubernetes.io/projected/f3c39857-81ae-4d39-a06d-95f4e115951b-kube-api-access-28vcs\") pod \"auth-test-model-361849-kserve-7d4cc4c487-m7bhf\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") " pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:36.981762 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:36.981698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:37.264201 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:37.264176    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"]
Sep 17 01:47:37.266128 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:47:37.266101    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3c39857_81ae_4d39_a06d_95f4e115951b.slice/crio-ee5a46e16d8050059de92407fb44e0f8d9b02847d87394619af1bc8ae8b6ce99 WatchSource:0}: Error finding container ee5a46e16d8050059de92407fb44e0f8d9b02847d87394619af1bc8ae8b6ce99: Status 404 returned error can't find the container with id ee5a46e16d8050059de92407fb44e0f8d9b02847d87394619af1bc8ae8b6ce99
Sep 17 01:47:37.696876 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:37.696840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf" event={"ID":"f3c39857-81ae-4d39-a06d-95f4e115951b","Type":"ContainerStarted","Data":"f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384"}
Sep 17 01:47:37.697243 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:37.696885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf" event={"ID":"f3c39857-81ae-4d39-a06d-95f4e115951b","Type":"ContainerStarted","Data":"ee5a46e16d8050059de92407fb44e0f8d9b02847d87394619af1bc8ae8b6ce99"}
Sep 17 01:47:37.713592 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:37.713520    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf" podStartSLOduration=1.713505923 podStartE2EDuration="1.713505923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:47:37.712003258 +0000 UTC m=+1536.382519352" watchObservedRunningTime="2026-09-17 01:47:37.713505923 +0000 UTC m=+1536.384022011"
Sep 17 01:47:46.982467 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:46.982427    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:46.982966 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:46.982577    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:46.995381 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:46.995358    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:47:47.757722 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:47:47.757692    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:48:28.463913 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.463877    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"]
Sep 17 01:48:28.464489 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.464149    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf" podUID="933c6748-c325-4639-b905-776f3102b63d" containerName="maas-api" containerID="cri-o://cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4" gracePeriod=30
Sep 17 01:48:28.523934 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.523900    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"]
Sep 17 01:48:28.524192 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.524160    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6" podUID="cdbc353f-9ddb-46e3-92ba-5379c9a826dd" containerName="payload-pre-processing" containerID="cri-o://3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13" gracePeriod=30
Sep 17 01:48:28.776443 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.776412    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:48:28.800196 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.800169    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:48:28.879720 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.879688    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-maas-api-tls\") pod \"933c6748-c325-4639-b905-776f3102b63d\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") "
Sep 17 01:48:28.879898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.879767    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933c6748-c325-4639-b905-776f3102b63d-openshift-service-ca\") pod \"933c6748-c325-4639-b905-776f3102b63d\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") "
Sep 17 01:48:28.879898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.879831    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dgpz\" (UniqueName: \"kubernetes.io/projected/933c6748-c325-4639-b905-776f3102b63d-kube-api-access-6dgpz\") pod \"933c6748-c325-4639-b905-776f3102b63d\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") "
Sep 17 01:48:28.880010 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.879959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-plugins-config-volume\") pod \"cdbc353f-9ddb-46e3-92ba-5379c9a826dd\" (UID: \"cdbc353f-9ddb-46e3-92ba-5379c9a826dd\") "
Sep 17 01:48:28.880066 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.880028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7vm2\" (UniqueName: \"kubernetes.io/projected/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-kube-api-access-s7vm2\") pod \"cdbc353f-9ddb-46e3-92ba-5379c9a826dd\" (UID: \"cdbc353f-9ddb-46e3-92ba-5379c9a826dd\") "
Sep 17 01:48:28.880116 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.880068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-metrics-certs\") pod \"933c6748-c325-4639-b905-776f3102b63d\" (UID: \"933c6748-c325-4639-b905-776f3102b63d\") "
Sep 17 01:48:28.880184 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.880122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933c6748-c325-4639-b905-776f3102b63d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "933c6748-c325-4639-b905-776f3102b63d" (UID: "933c6748-c325-4639-b905-776f3102b63d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:48:28.880352 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.880327    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cdbc353f-9ddb-46e3-92ba-5379c9a826dd" (UID: "cdbc353f-9ddb-46e3-92ba-5379c9a826dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:48:28.880580 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.880528    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-plugins-config-volume\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:28.880580 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.880580    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933c6748-c325-4639-b905-776f3102b63d-openshift-service-ca\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:28.882024 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.881999    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "933c6748-c325-4639-b905-776f3102b63d" (UID: "933c6748-c325-4639-b905-776f3102b63d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:48:28.882024 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.882014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933c6748-c325-4639-b905-776f3102b63d-kube-api-access-6dgpz" (OuterVolumeSpecName: "kube-api-access-6dgpz") pod "933c6748-c325-4639-b905-776f3102b63d" (UID: "933c6748-c325-4639-b905-776f3102b63d"). InnerVolumeSpecName "kube-api-access-6dgpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:48:28.882569 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.882550    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "933c6748-c325-4639-b905-776f3102b63d" (UID: "933c6748-c325-4639-b905-776f3102b63d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:48:28.882636 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.882614    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-kube-api-access-s7vm2" (OuterVolumeSpecName: "kube-api-access-s7vm2") pod "cdbc353f-9ddb-46e3-92ba-5379c9a826dd" (UID: "cdbc353f-9ddb-46e3-92ba-5379c9a826dd"). InnerVolumeSpecName "kube-api-access-s7vm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:48:28.962925 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.962887    2578 generic.go:356] "Generic (PLEG): container finished" podID="933c6748-c325-4639-b905-776f3102b63d" containerID="cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4" exitCode=0
Sep 17 01:48:28.963088 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.962938    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"
Sep 17 01:48:28.963088 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.962966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf" event={"ID":"933c6748-c325-4639-b905-776f3102b63d","Type":"ContainerDied","Data":"cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4"}
Sep 17 01:48:28.963088 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.963021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf" event={"ID":"933c6748-c325-4639-b905-776f3102b63d","Type":"ContainerDied","Data":"c28b1e3f5a832a0b88dd8247829b2a9d7326ecc2975f291145aec87d6c956e4e"}
Sep 17 01:48:28.963088 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.963042    2578 scope.go:117] "RemoveContainer" containerID="cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4"
Sep 17 01:48:28.964506 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.964457    2578 generic.go:356] "Generic (PLEG): container finished" podID="cdbc353f-9ddb-46e3-92ba-5379c9a826dd" containerID="3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13" exitCode=0
Sep 17 01:48:28.964506 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.964481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6" event={"ID":"cdbc353f-9ddb-46e3-92ba-5379c9a826dd","Type":"ContainerDied","Data":"3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13"}
Sep 17 01:48:28.964725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.964504    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"
Sep 17 01:48:28.964725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.964530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6" event={"ID":"cdbc353f-9ddb-46e3-92ba-5379c9a826dd","Type":"ContainerDied","Data":"08a9c98717f3f66e617f91dc75066b21399f490741f4936da7a52b1d295db900"}
Sep 17 01:48:28.972320 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.972298    2578 scope.go:117] "RemoveContainer" containerID="cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4"
Sep 17 01:48:28.972686 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:48:28.972667    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4\": container with ID starting with cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4 not found: ID does not exist" containerID="cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4"
Sep 17 01:48:28.972782 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.972700    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4"} err="failed to get container status \"cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4\": rpc error: code = NotFound desc = could not find container \"cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4\": container with ID starting with cd4664cc468fd4057a33fad4b3b04b9029a3bb10ddcfa7c75feb21f91a9d8ac4 not found: ID does not exist"
Sep 17 01:48:28.972782 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.972723    2578 scope.go:117] "RemoveContainer" containerID="3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13"
Sep 17 01:48:28.981260 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.981242    2578 scope.go:117] "RemoveContainer" containerID="3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13"
Sep 17 01:48:28.981602 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:48:28.981578    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13\": container with ID starting with 3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13 not found: ID does not exist" containerID="3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13"
Sep 17 01:48:28.981662 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.981611    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13"} err="failed to get container status \"3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13\": rpc error: code = NotFound desc = could not find container \"3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13\": container with ID starting with 3f3c4336877d069a9affe4289cb6ffba05ea756e530aa7bcfdbc2b77b0662e13 not found: ID does not exist"
Sep 17 01:48:28.981804 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.981788    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dgpz\" (UniqueName: \"kubernetes.io/projected/933c6748-c325-4639-b905-776f3102b63d-kube-api-access-6dgpz\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:28.981838 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.981808    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7vm2\" (UniqueName: \"kubernetes.io/projected/cdbc353f-9ddb-46e3-92ba-5379c9a826dd-kube-api-access-s7vm2\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:28.981838 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.981820    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-metrics-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:28.981838 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.981828    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933c6748-c325-4639-b905-776f3102b63d-maas-api-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:28.987375 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.987350    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"]
Sep 17 01:48:28.993594 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:28.992580    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a1db100-5867d996b6-kq5s6"]
Sep 17 01:48:29.002747 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:29.002717    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"]
Sep 17 01:48:29.006161 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:29.006132    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a1db100-778c65bd8-nmqmf"]
Sep 17 01:48:29.920930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:29.920897    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="933c6748-c325-4639-b905-776f3102b63d" path="/var/lib/kubelet/pods/933c6748-c325-4639-b905-776f3102b63d/volumes"
Sep 17 01:48:29.921317 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:29.921264    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdbc353f-9ddb-46e3-92ba-5379c9a826dd" path="/var/lib/kubelet/pods/cdbc353f-9ddb-46e3-92ba-5379c9a826dd/volumes"
Sep 17 01:48:36.840446 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:36.840408    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"]
Sep 17 01:48:51.860363 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:51.860306    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw" podUID="b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" containerName="main" containerID="cri-o://dd5d3758046bbdfbdde8e948bd7d3870123e09bd723c9bbb3b806963d9a602b6" gracePeriod=45
Sep 17 01:48:52.083607 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.083571    2578 generic.go:356] "Generic (PLEG): container finished" podID="b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" containerID="dd5d3758046bbdfbdde8e948bd7d3870123e09bd723c9bbb3b806963d9a602b6" exitCode=0
Sep 17 01:48:52.083760 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.083624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw" event={"ID":"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4","Type":"ContainerDied","Data":"dd5d3758046bbdfbdde8e948bd7d3870123e09bd723c9bbb3b806963d9a602b6"}
Sep 17 01:48:52.115225 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.115171    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:48:52.220471 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.220444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-dshm\") pod \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") "
Sep 17 01:48:52.220663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.220489    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85stv\" (UniqueName: \"kubernetes.io/projected/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-kube-api-access-85stv\") pod \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") "
Sep 17 01:48:52.220663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.220511    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tls-certs\") pod \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") "
Sep 17 01:48:52.220663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.220595    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-home\") pod \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") "
Sep 17 01:48:52.220663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.220663    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-model-cache\") pod \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") "
Sep 17 01:48:52.220903 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.220688    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tmp-dir\") pod \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\" (UID: \"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4\") "
Sep 17 01:48:52.221082 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.221054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" (UID: "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:48:52.221145 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.221133    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-home" (OuterVolumeSpecName: "home") pod "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" (UID: "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:48:52.221200 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.221170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" (UID: "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:48:52.222786 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.222755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-dshm" (OuterVolumeSpecName: "dshm") pod "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" (UID: "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:48:52.222894 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.222822    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" (UID: "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:48:52.222894 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.222872    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-kube-api-access-85stv" (OuterVolumeSpecName: "kube-api-access-85stv") pod "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" (UID: "b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4"). InnerVolumeSpecName "kube-api-access-85stv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:48:52.322328 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.322296    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:52.322328 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.322322    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:52.322328 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.322332    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:52.322572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.322340    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:52.322572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.322347    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:52.322572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:52.322354    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85stv\" (UniqueName: \"kubernetes.io/projected/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4-kube-api-access-85stv\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:48:53.089517 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:53.089493    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"
Sep 17 01:48:53.090024 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:53.089554    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw" event={"ID":"b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4","Type":"ContainerDied","Data":"2cfcf0e2a2bae1480ba569bb1463037f9dad23090d5dc10f9d500f7e1858c3ca"}
Sep 17 01:48:53.090024 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:53.089605    2578 scope.go:117] "RemoveContainer" containerID="dd5d3758046bbdfbdde8e948bd7d3870123e09bd723c9bbb3b806963d9a602b6"
Sep 17 01:48:53.109825 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:53.109801    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"]
Sep 17 01:48:53.113352 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:53.113330    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2a1db100/e2e-lifecycle-model-2a1db100-kserve-db99b6fcf-mqjsw"]
Sep 17 01:48:53.921444 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:48:53.921363    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" path="/var/lib/kubelet/pods/b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4/volumes"
Sep 17 01:49:11.576982 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:11.576950    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"]
Sep 17 01:49:11.637502 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:11.637467    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"]
Sep 17 01:49:22.486275 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.486241    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"]
Sep 17 01:49:22.486771 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.486511    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78" podUID="7f500283-725c-4a50-8264-cce6604c0b7f" containerName="maas-api" containerID="cri-o://1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84" gracePeriod=30
Sep 17 01:49:22.736924 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.736864    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:49:22.823887 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.823850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzsrc\" (UniqueName: \"kubernetes.io/projected/7f500283-725c-4a50-8264-cce6604c0b7f-kube-api-access-mzsrc\") pod \"7f500283-725c-4a50-8264-cce6604c0b7f\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") "
Sep 17 01:49:22.824072 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.823907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f500283-725c-4a50-8264-cce6604c0b7f-openshift-service-ca\") pod \"7f500283-725c-4a50-8264-cce6604c0b7f\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") "
Sep 17 01:49:22.824072 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.823959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-metrics-certs\") pod \"7f500283-725c-4a50-8264-cce6604c0b7f\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") "
Sep 17 01:49:22.824072 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.824047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-maas-api-tls\") pod \"7f500283-725c-4a50-8264-cce6604c0b7f\" (UID: \"7f500283-725c-4a50-8264-cce6604c0b7f\") "
Sep 17 01:49:22.824366 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.824339    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f500283-725c-4a50-8264-cce6604c0b7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f500283-725c-4a50-8264-cce6604c0b7f" (UID: "7f500283-725c-4a50-8264-cce6604c0b7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:49:22.824471 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.824451    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f500283-725c-4a50-8264-cce6604c0b7f-openshift-service-ca\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:22.826125 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.826099    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f500283-725c-4a50-8264-cce6604c0b7f-kube-api-access-mzsrc" (OuterVolumeSpecName: "kube-api-access-mzsrc") pod "7f500283-725c-4a50-8264-cce6604c0b7f" (UID: "7f500283-725c-4a50-8264-cce6604c0b7f"). InnerVolumeSpecName "kube-api-access-mzsrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:49:22.826237 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.826125    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7f500283-725c-4a50-8264-cce6604c0b7f" (UID: "7f500283-725c-4a50-8264-cce6604c0b7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:49:22.826237 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.826141    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f500283-725c-4a50-8264-cce6604c0b7f" (UID: "7f500283-725c-4a50-8264-cce6604c0b7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:49:22.925212 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.925182    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-metrics-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:22.925212 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.925206    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f500283-725c-4a50-8264-cce6604c0b7f-maas-api-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:22.925212 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:22.925216    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzsrc\" (UniqueName: \"kubernetes.io/projected/7f500283-725c-4a50-8264-cce6604c0b7f-kube-api-access-mzsrc\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:23.237214 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:23.237176    2578 generic.go:356] "Generic (PLEG): container finished" podID="7f500283-725c-4a50-8264-cce6604c0b7f" containerID="1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84" exitCode=0
Sep 17 01:49:23.237214 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:23.237217    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"
Sep 17 01:49:23.237456 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:23.237218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78" event={"ID":"7f500283-725c-4a50-8264-cce6604c0b7f","Type":"ContainerDied","Data":"1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84"}
Sep 17 01:49:23.237456 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:23.237353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78" event={"ID":"7f500283-725c-4a50-8264-cce6604c0b7f","Type":"ContainerDied","Data":"65c7baf8fef577d437f1155eb2505eaf535555a45a59eeb7e0ce79a0992d39ae"}
Sep 17 01:49:23.237456 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:23.237381    2578 scope.go:117] "RemoveContainer" containerID="1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84"
Sep 17 01:49:23.247725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:23.247704    2578 scope.go:117] "RemoveContainer" containerID="1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84"
Sep 17 01:49:23.248030 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:49:23.248015    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84\": container with ID starting with 1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84 not found: ID does not exist" containerID="1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84"
Sep 17 01:49:23.248079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:23.248038    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84"} err="failed to get container status \"1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84\": rpc error: code = NotFound desc = could not find container \"1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84\": container with ID starting with 1da2efcad3ceb635e899eed253b79d3a983d0b7622602c9e462b0eceba1a8e84 not found: ID does not exist"
Sep 17 01:49:23.271671 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:23.271637    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"]
Sep 17 01:49:23.280014 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:23.279984    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8be734-68f957b4f7-9ll78"]
Sep 17 01:49:23.920903 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:23.920868    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f500283-725c-4a50-8264-cce6604c0b7f" path="/var/lib/kubelet/pods/7f500283-725c-4a50-8264-cce6604c0b7f/volumes"
Sep 17 01:49:26.595935 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:26.595888    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw" podUID="94a55a27-78c1-4cf7-ba82-00034f830d99" containerName="main" containerID="cri-o://3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0" gracePeriod=45
Sep 17 01:49:26.656471 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:26.656433    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c" podUID="97a445cd-d987-46bb-b3b7-e5495d1ebb3d" containerName="main" containerID="cri-o://2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189" gracePeriod=45
Sep 17 01:49:26.786004 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:26.785960    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw" podUID="94a55a27-78c1-4cf7-ba82-00034f830d99" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 01:49:26.921867 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:26.921836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:49:26.923963 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:26.923938    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:49:27.064474 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064438    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftwvv\" (UniqueName: \"kubernetes.io/projected/94a55a27-78c1-4cf7-ba82-00034f830d99-kube-api-access-ftwvv\") pod \"94a55a27-78c1-4cf7-ba82-00034f830d99\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") "
Sep 17 01:49:27.064704 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-home\") pod \"94a55a27-78c1-4cf7-ba82-00034f830d99\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") "
Sep 17 01:49:27.064704 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84d8s\" (UniqueName: \"kubernetes.io/projected/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-kube-api-access-84d8s\") pod \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") "
Sep 17 01:49:27.064704 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-model-cache\") pod \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") "
Sep 17 01:49:27.064704 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064618    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-home\") pod \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") "
Sep 17 01:49:27.064930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-dshm\") pod \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") "
Sep 17 01:49:27.064930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064787    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-home" (OuterVolumeSpecName: "home") pod "94a55a27-78c1-4cf7-ba82-00034f830d99" (UID: "94a55a27-78c1-4cf7-ba82-00034f830d99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:49:27.064930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064807    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-model-cache\") pod \"94a55a27-78c1-4cf7-ba82-00034f830d99\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") "
Sep 17 01:49:27.064930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064843    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-home" (OuterVolumeSpecName: "home") pod "97a445cd-d987-46bb-b3b7-e5495d1ebb3d" (UID: "97a445cd-d987-46bb-b3b7-e5495d1ebb3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:49:27.064930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "97a445cd-d987-46bb-b3b7-e5495d1ebb3d" (UID: "97a445cd-d987-46bb-b3b7-e5495d1ebb3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:49:27.064930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064864    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tmp-dir\") pod \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") "
Sep 17 01:49:27.064930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a55a27-78c1-4cf7-ba82-00034f830d99-tls-certs\") pod \"94a55a27-78c1-4cf7-ba82-00034f830d99\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") "
Sep 17 01:49:27.065348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064939    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tls-certs\") pod \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\" (UID: \"97a445cd-d987-46bb-b3b7-e5495d1ebb3d\") "
Sep 17 01:49:27.065348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.064986    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-dshm\") pod \"94a55a27-78c1-4cf7-ba82-00034f830d99\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") "
Sep 17 01:49:27.065348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.065044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97a445cd-d987-46bb-b3b7-e5495d1ebb3d" (UID: "97a445cd-d987-46bb-b3b7-e5495d1ebb3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:49:27.065348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.065046    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-model-cache" (OuterVolumeSpecName: "model-cache") pod "94a55a27-78c1-4cf7-ba82-00034f830d99" (UID: "94a55a27-78c1-4cf7-ba82-00034f830d99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:49:27.065348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.065088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-tmp-dir\") pod \"94a55a27-78c1-4cf7-ba82-00034f830d99\" (UID: \"94a55a27-78c1-4cf7-ba82-00034f830d99\") "
Sep 17 01:49:27.065642 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.065563    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.065642 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.065584    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.065642 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.065599    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.065642 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.065603    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94a55a27-78c1-4cf7-ba82-00034f830d99" (UID: "94a55a27-78c1-4cf7-ba82-00034f830d99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:49:27.065642 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.065612    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.065898 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.065657    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.067183 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.067148    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97a445cd-d987-46bb-b3b7-e5495d1ebb3d" (UID: "97a445cd-d987-46bb-b3b7-e5495d1ebb3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:49:27.067183 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.067164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-kube-api-access-84d8s" (OuterVolumeSpecName: "kube-api-access-84d8s") pod "97a445cd-d987-46bb-b3b7-e5495d1ebb3d" (UID: "97a445cd-d987-46bb-b3b7-e5495d1ebb3d"). InnerVolumeSpecName "kube-api-access-84d8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:49:27.067314 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.067251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a55a27-78c1-4cf7-ba82-00034f830d99-kube-api-access-ftwvv" (OuterVolumeSpecName: "kube-api-access-ftwvv") pod "94a55a27-78c1-4cf7-ba82-00034f830d99" (UID: "94a55a27-78c1-4cf7-ba82-00034f830d99"). InnerVolumeSpecName "kube-api-access-ftwvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:49:27.067426 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.067405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-dshm" (OuterVolumeSpecName: "dshm") pod "97a445cd-d987-46bb-b3b7-e5495d1ebb3d" (UID: "97a445cd-d987-46bb-b3b7-e5495d1ebb3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:49:27.067468 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.067408    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-dshm" (OuterVolumeSpecName: "dshm") pod "94a55a27-78c1-4cf7-ba82-00034f830d99" (UID: "94a55a27-78c1-4cf7-ba82-00034f830d99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:49:27.067716 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.067694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a55a27-78c1-4cf7-ba82-00034f830d99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94a55a27-78c1-4cf7-ba82-00034f830d99" (UID: "94a55a27-78c1-4cf7-ba82-00034f830d99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:49:27.166957 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.166870    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.166957 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.166901    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftwvv\" (UniqueName: \"kubernetes.io/projected/94a55a27-78c1-4cf7-ba82-00034f830d99-kube-api-access-ftwvv\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.166957 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.166912    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84d8s\" (UniqueName: \"kubernetes.io/projected/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-kube-api-access-84d8s\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.166957 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.166921    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.166957 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.166930    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a55a27-78c1-4cf7-ba82-00034f830d99-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.166957 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.166938    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a445cd-d987-46bb-b3b7-e5495d1ebb3d-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.166957 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.166945    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a55a27-78c1-4cf7-ba82-00034f830d99-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:49:27.259349 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.259309    2578 generic.go:356] "Generic (PLEG): container finished" podID="94a55a27-78c1-4cf7-ba82-00034f830d99" containerID="3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0" exitCode=0
Sep 17 01:49:27.259560 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.259357    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"
Sep 17 01:49:27.259560 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.259392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw" event={"ID":"94a55a27-78c1-4cf7-ba82-00034f830d99","Type":"ContainerDied","Data":"3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0"}
Sep 17 01:49:27.259560 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.259434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw" event={"ID":"94a55a27-78c1-4cf7-ba82-00034f830d99","Type":"ContainerDied","Data":"7647be581d5c94ede1c5ec25ee9a96eaa25524effae49803ab0e7ec7071289bb"}
Sep 17 01:49:27.259560 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.259457    2578 scope.go:117] "RemoveContainer" containerID="3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0"
Sep 17 01:49:27.261044 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.261023    2578 generic.go:356] "Generic (PLEG): container finished" podID="97a445cd-d987-46bb-b3b7-e5495d1ebb3d" containerID="2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189" exitCode=0
Sep 17 01:49:27.261164 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.261062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c" event={"ID":"97a445cd-d987-46bb-b3b7-e5495d1ebb3d","Type":"ContainerDied","Data":"2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189"}
Sep 17 01:49:27.261164 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.261080    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"
Sep 17 01:49:27.261164 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.261085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c" event={"ID":"97a445cd-d987-46bb-b3b7-e5495d1ebb3d","Type":"ContainerDied","Data":"428bce6661ef8521855e7f585073f1d1beed38e3bbf4ab9468be82cf531ca45e"}
Sep 17 01:49:27.268199 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.268183    2578 scope.go:117] "RemoveContainer" containerID="3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0"
Sep 17 01:49:27.268441 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:49:27.268425    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0\": container with ID starting with 3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0 not found: ID does not exist" containerID="3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0"
Sep 17 01:49:27.268486 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.268448    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0"} err="failed to get container status \"3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0\": rpc error: code = NotFound desc = could not find container \"3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0\": container with ID starting with 3bbadb197ed38f32d6a0a8d50ae9fe6d633c20f97f6ce7f3be9e7fc9945513c0 not found: ID does not exist"
Sep 17 01:49:27.268486 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.268464    2578 scope.go:117] "RemoveContainer" containerID="2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189"
Sep 17 01:49:27.276320 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.276305    2578 scope.go:117] "RemoveContainer" containerID="2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189"
Sep 17 01:49:27.276594 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:49:27.276575    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189\": container with ID starting with 2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189 not found: ID does not exist" containerID="2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189"
Sep 17 01:49:27.276658 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.276602    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189"} err="failed to get container status \"2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189\": rpc error: code = NotFound desc = could not find container \"2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189\": container with ID starting with 2d5f086f983497b5fdc9c6a69c0b3ca8873259302ce725a2cf41c2ebe080a189 not found: ID does not exist"
Sep 17 01:49:27.283770 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.283749    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"]
Sep 17 01:49:27.287449 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.287428    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/distinct-2-w0-8be734-kserve-6b4cb49765-5m4xw"]
Sep 17 01:49:27.297018 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.296990    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"]
Sep 17 01:49:27.300406 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.300387    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/simulator-w0-8be734-kserve-b477c4bf4-7dt9c"]
Sep 17 01:49:27.926345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.922893    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94a55a27-78c1-4cf7-ba82-00034f830d99" path="/var/lib/kubelet/pods/94a55a27-78c1-4cf7-ba82-00034f830d99/volumes"
Sep 17 01:49:27.926345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:49:27.923508    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a445cd-d987-46bb-b3b7-e5495d1ebb3d" path="/var/lib/kubelet/pods/97a445cd-d987-46bb-b3b7-e5495d1ebb3d/volumes"
Sep 17 01:50:15.836938 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.836908    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"]
Sep 17 01:50:15.837348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837279    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a445cd-d987-46bb-b3b7-e5495d1ebb3d" containerName="main"
Sep 17 01:50:15.837348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837292    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a445cd-d987-46bb-b3b7-e5495d1ebb3d" containerName="main"
Sep 17 01:50:15.837348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837302    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="933c6748-c325-4639-b905-776f3102b63d" containerName="maas-api"
Sep 17 01:50:15.837348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837307    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="933c6748-c325-4639-b905-776f3102b63d" containerName="maas-api"
Sep 17 01:50:15.837348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837325    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" containerName="main"
Sep 17 01:50:15.837348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837330    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" containerName="main"
Sep 17 01:50:15.837348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837337    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f500283-725c-4a50-8264-cce6604c0b7f" containerName="maas-api"
Sep 17 01:50:15.837348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837342    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f500283-725c-4a50-8264-cce6604c0b7f" containerName="maas-api"
Sep 17 01:50:15.837348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837348    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a55a27-78c1-4cf7-ba82-00034f830d99" containerName="main"
Sep 17 01:50:15.837348 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837353    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a55a27-78c1-4cf7-ba82-00034f830d99" containerName="main"
Sep 17 01:50:15.837675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837360    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbc353f-9ddb-46e3-92ba-5379c9a826dd" containerName="payload-pre-processing"
Sep 17 01:50:15.837675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837367    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbc353f-9ddb-46e3-92ba-5379c9a826dd" containerName="payload-pre-processing"
Sep 17 01:50:15.837675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837420    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbc353f-9ddb-46e3-92ba-5379c9a826dd" containerName="payload-pre-processing"
Sep 17 01:50:15.837675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837429    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="933c6748-c325-4639-b905-776f3102b63d" containerName="maas-api"
Sep 17 01:50:15.837675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837436    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a55a27-78c1-4cf7-ba82-00034f830d99" containerName="main"
Sep 17 01:50:15.837675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837442    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a445cd-d987-46bb-b3b7-e5495d1ebb3d" containerName="main"
Sep 17 01:50:15.837675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837447    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a3414e-1c6e-4862-8bbf-86c3daf4c7b4" containerName="main"
Sep 17 01:50:15.837675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.837453    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f500283-725c-4a50-8264-cce6604c0b7f" containerName="maas-api"
Sep 17 01:50:15.840567 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.840523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:15.842875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.842854    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1cf28c-openshift-default-dockercfg-c4gl5\""
Sep 17 01:50:15.843005 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.842855    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 01:50:15.855688 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.855655    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"]
Sep 17 01:50:15.937056 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.937018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd0a697-a92e-4b5d-860a-93836be86cd0-istiod-ca-cert\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:15.937255 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.937073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-podinfo\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:15.937255 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.937093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-credential-socket\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:15.937255 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.937128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-socket\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:15.937255 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.937154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-envoy\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:15.937255 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.937224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-certs\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:15.937515 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.937265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7hpl\" (UniqueName: \"kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-kube-api-access-x7hpl\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:15.937515 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.937284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-token\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:15.937515 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:15.937309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-data\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038029 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.037998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-credential-socket\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038211 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-socket\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038211 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-envoy\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038211 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-certs\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038211 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7hpl\" (UniqueName: \"kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-kube-api-access-x7hpl\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038421 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-token\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038421 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-data\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038421 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd0a697-a92e-4b5d-860a-93836be86cd0-istiod-ca-cert\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038421 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-podinfo\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038668 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-credential-socket\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038668 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-socket\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038668 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-certs\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.038823 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.038674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-data\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.039094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.039071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd0a697-a92e-4b5d-860a-93836be86cd0-istiod-ca-cert\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.040705 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.040687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-podinfo\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.041056 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.041023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-envoy\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.046434 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.046412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-token\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.046522 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.046441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7hpl\" (UniqueName: \"kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-kube-api-access-x7hpl\") pod \"e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") " pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.161570 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.161446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:16.434954 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.434878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"]
Sep 17 01:50:16.438266 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:50:16.438234    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd0a697_a92e_4b5d_860a_93836be86cd0.slice/crio-43959e60c8b935810cda01f80adf816c9bdae4665f20b553bfbde72e652d2354 WatchSource:0}: Error finding container 43959e60c8b935810cda01f80adf816c9bdae4665f20b553bfbde72e652d2354: Status 404 returned error can't find the container with id 43959e60c8b935810cda01f80adf816c9bdae4665f20b553bfbde72e652d2354
Sep 17 01:50:16.493905 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:16.493869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" event={"ID":"ddd0a697-a92e-4b5d-860a-93836be86cd0","Type":"ContainerStarted","Data":"43959e60c8b935810cda01f80adf816c9bdae4665f20b553bfbde72e652d2354"}
Sep 17 01:50:19.787761 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:19.787724    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:50:19.788036 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:19.787799    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:50:19.788036 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:19.787827    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:50:20.514858 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:20.514822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" event={"ID":"ddd0a697-a92e-4b5d-860a-93836be86cd0","Type":"ContainerStarted","Data":"90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274"}
Sep 17 01:50:20.537171 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:20.537100    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" podStartSLOduration=2.190508586 podStartE2EDuration="5.537073445s" totalImagesPullingTime="3.346564859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:20.533068761 +0000 UTC m=+1699.203584851" watchObservedRunningTime="2026-09-17 01:50:20.537073445 +0000 UTC m=+1699.207589535"
Sep 17 01:50:21.162649 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:21.162612    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:21.167768 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:21.167743    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:21.519704 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:21.519621    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:21.520499 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:21.520479    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:25.060676 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.060562    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"]
Sep 17 01:50:25.065452 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.065430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.067719 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.067699    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1cf28c-serving-cert\""
Sep 17 01:50:25.075137 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.075111    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"]
Sep 17 01:50:25.101258 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.101224    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"]
Sep 17 01:50:25.126945 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.126916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-metrics-certs\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.127066 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.126974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r699r\" (UniqueName: \"kubernetes.io/projected/d0299d70-4e8a-42be-a620-d1f344ceaaa2-kube-api-access-r699r\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.127066 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.127028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-maas-api-tls\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.127146 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.127098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0299d70-4e8a-42be-a620-d1f344ceaaa2-openshift-service-ca\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.227900 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.227872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-metrics-certs\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.228051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.227918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r699r\" (UniqueName: \"kubernetes.io/projected/d0299d70-4e8a-42be-a620-d1f344ceaaa2-kube-api-access-r699r\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.228051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.227974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-maas-api-tls\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.228051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.228006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0299d70-4e8a-42be-a620-d1f344ceaaa2-openshift-service-ca\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.228185 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:50:25.228088    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-serving-cert: secret "maas-api-e2e-api-a-1cf28c-serving-cert" not found
Sep 17 01:50:25.228185 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:50:25.228164    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-maas-api-tls podName:d0299d70-4e8a-42be-a620-d1f344ceaaa2 nodeName:}" failed. No retries permitted until 2026-09-17 01:50:25.728137247 +0000 UTC m=+1704.398653332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-maas-api-tls") pod "maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7" (UID: "d0299d70-4e8a-42be-a620-d1f344ceaaa2") : secret "maas-api-e2e-api-a-1cf28c-serving-cert" not found
Sep 17 01:50:25.228695 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.228677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0299d70-4e8a-42be-a620-d1f344ceaaa2-openshift-service-ca\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.230196 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.230177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-metrics-certs\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.236871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.236842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r699r\" (UniqueName: \"kubernetes.io/projected/d0299d70-4e8a-42be-a620-d1f344ceaaa2-kube-api-access-r699r\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.667181 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.667151    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"]
Sep 17 01:50:25.670937 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.670917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:50:25.673282 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.673263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1cf28c\""
Sep 17 01:50:25.673380 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.673291    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1cf28c-dockercfg-5sth9\""
Sep 17 01:50:25.683027 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.683002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"]
Sep 17 01:50:25.708829 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.708798    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"]
Sep 17 01:50:25.712776 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.712758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:50:25.720736 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.720712    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"]
Sep 17 01:50:25.738434 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.738412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-maas-api-tls\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.738566 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.738491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec56441a-baee-487c-914d-6bec7d2107c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz\" (UID: \"ec56441a-baee-487c-914d-6bec7d2107c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:50:25.738566 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.738560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tc24\" (UniqueName: \"kubernetes.io/projected/ec56441a-baee-487c-914d-6bec7d2107c0-kube-api-access-6tc24\") pod \"payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz\" (UID: \"ec56441a-baee-487c-914d-6bec7d2107c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:50:25.738660 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.738625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a7224c-1a69-4509-888f-46d9a0f77553-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt\" (UID: \"37a7224c-1a69-4509-888f-46d9a0f77553\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:50:25.738697 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.738663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59dx\" (UniqueName: \"kubernetes.io/projected/37a7224c-1a69-4509-888f-46d9a0f77553-kube-api-access-f59dx\") pod \"payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt\" (UID: \"37a7224c-1a69-4509-888f-46d9a0f77553\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:50:25.740744 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.740724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-maas-api-tls\") pod \"maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.839555 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.839493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tc24\" (UniqueName: \"kubernetes.io/projected/ec56441a-baee-487c-914d-6bec7d2107c0-kube-api-access-6tc24\") pod \"payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz\" (UID: \"ec56441a-baee-487c-914d-6bec7d2107c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:50:25.839733 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.839581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a7224c-1a69-4509-888f-46d9a0f77553-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt\" (UID: \"37a7224c-1a69-4509-888f-46d9a0f77553\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:50:25.839733 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.839606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59dx\" (UniqueName: \"kubernetes.io/projected/37a7224c-1a69-4509-888f-46d9a0f77553-kube-api-access-f59dx\") pod \"payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt\" (UID: \"37a7224c-1a69-4509-888f-46d9a0f77553\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:50:25.839856 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.839762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec56441a-baee-487c-914d-6bec7d2107c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz\" (UID: \"ec56441a-baee-487c-914d-6bec7d2107c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:50:25.840297 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.840271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a7224c-1a69-4509-888f-46d9a0f77553-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt\" (UID: \"37a7224c-1a69-4509-888f-46d9a0f77553\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:50:25.840495 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.840475    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec56441a-baee-487c-914d-6bec7d2107c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz\" (UID: \"ec56441a-baee-487c-914d-6bec7d2107c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:50:25.846963 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.846941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59dx\" (UniqueName: \"kubernetes.io/projected/37a7224c-1a69-4509-888f-46d9a0f77553-kube-api-access-f59dx\") pod \"payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt\" (UID: \"37a7224c-1a69-4509-888f-46d9a0f77553\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:50:25.847036 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.846963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tc24\" (UniqueName: \"kubernetes.io/projected/ec56441a-baee-487c-914d-6bec7d2107c0-kube-api-access-6tc24\") pod \"payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz\" (UID: \"ec56441a-baee-487c-914d-6bec7d2107c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:50:25.978744 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.978669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:25.983637 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:25.983617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:50:26.023724 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.023687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:50:26.262424 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.262386    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"]
Sep 17 01:50:26.262932 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:50:26.262874    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec56441a_baee_487c_914d_6bec7d2107c0.slice/crio-2d7a96b4bf5a8d48834c946fb0fdce649d40a547adfede46eaedcdb9ec47e8c7 WatchSource:0}: Error finding container 2d7a96b4bf5a8d48834c946fb0fdce649d40a547adfede46eaedcdb9ec47e8c7: Status 404 returned error can't find the container with id 2d7a96b4bf5a8d48834c946fb0fdce649d40a547adfede46eaedcdb9ec47e8c7
Sep 17 01:50:26.265026 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.265006    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"]
Sep 17 01:50:26.265394 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:50:26.265374    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0299d70_4e8a_42be_a620_d1f344ceaaa2.slice/crio-1790e4f1926707d19c6507b84b8851c9799bb3a37f14a630c1b78fe18d85ce82 WatchSource:0}: Error finding container 1790e4f1926707d19c6507b84b8851c9799bb3a37f14a630c1b78fe18d85ce82: Status 404 returned error can't find the container with id 1790e4f1926707d19c6507b84b8851c9799bb3a37f14a630c1b78fe18d85ce82
Sep 17 01:50:26.310116 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.310093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"]
Sep 17 01:50:26.315601 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:50:26.315573    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a7224c_1a69_4509_888f_46d9a0f77553.slice/crio-2f882957509acdee6dfa2f34e1758610a33c9e5381731d02c7604863e3403f73 WatchSource:0}: Error finding container 2f882957509acdee6dfa2f34e1758610a33c9e5381731d02c7604863e3403f73: Status 404 returned error can't find the container with id 2f882957509acdee6dfa2f34e1758610a33c9e5381731d02c7604863e3403f73
Sep 17 01:50:26.547065 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.547038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt" event={"ID":"37a7224c-1a69-4509-888f-46d9a0f77553","Type":"ContainerStarted","Data":"96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5"}
Sep 17 01:50:26.547248 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.547088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt" event={"ID":"37a7224c-1a69-4509-888f-46d9a0f77553","Type":"ContainerStarted","Data":"2f882957509acdee6dfa2f34e1758610a33c9e5381731d02c7604863e3403f73"}
Sep 17 01:50:26.547248 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.547143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:50:26.548854 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.548826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz" event={"ID":"ec56441a-baee-487c-914d-6bec7d2107c0","Type":"ContainerStarted","Data":"85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff"}
Sep 17 01:50:26.548963 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.548870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz" event={"ID":"ec56441a-baee-487c-914d-6bec7d2107c0","Type":"ContainerStarted","Data":"2d7a96b4bf5a8d48834c946fb0fdce649d40a547adfede46eaedcdb9ec47e8c7"}
Sep 17 01:50:26.548963 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.548904    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:50:26.550119 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.550097    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7" event={"ID":"d0299d70-4e8a-42be-a620-d1f344ceaaa2","Type":"ContainerStarted","Data":"1790e4f1926707d19c6507b84b8851c9799bb3a37f14a630c1b78fe18d85ce82"}
Sep 17 01:50:26.576371 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.576321    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt" podStartSLOduration=1.576303179 podStartE2EDuration="1.576303179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:26.575049414 +0000 UTC m=+1705.245565495" watchObservedRunningTime="2026-09-17 01:50:26.576303179 +0000 UTC m=+1705.246819269"
Sep 17 01:50:26.596463 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:26.596417    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz" podStartSLOduration=1.596346772 podStartE2EDuration="1.596346772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:26.595614657 +0000 UTC m=+1705.266130748" watchObservedRunningTime="2026-09-17 01:50:26.596346772 +0000 UTC m=+1705.266862866"
Sep 17 01:50:27.557090 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:27.557054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7" event={"ID":"d0299d70-4e8a-42be-a620-d1f344ceaaa2","Type":"ContainerStarted","Data":"32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2"}
Sep 17 01:50:27.557471 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:27.557383    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:27.576074 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:27.576023    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7" podStartSLOduration=2.2408004 podStartE2EDuration="2.576007163s" totalImagesPullingTime="335.206763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:27.574171528 +0000 UTC m=+1706.244687620" watchObservedRunningTime="2026-09-17 01:50:27.576007163 +0000 UTC m=+1706.246523670"
Sep 17 01:50:27.650284 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:27.650251    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"]
Sep 17 01:50:33.573882 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:33.573844    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:50:36.159919 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.159888    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"]
Sep 17 01:50:36.163480 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.163460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.165989 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.165966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-fcd402\"/\"sub-test-model-fcd402-kserve-self-signed-certs\""
Sep 17 01:50:36.181910 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.173107    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"]
Sep 17 01:50:36.241464 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.241433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-model-cache\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.241639 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.241471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzx6z\" (UniqueName: \"kubernetes.io/projected/6ad0d16e-c06e-4341-8cb4-51908394bf61-kube-api-access-zzx6z\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.241639 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.241588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-home\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.241639 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.241618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-dshm\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.241752 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.241649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad0d16e-c06e-4341-8cb4-51908394bf61-tls-certs\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.241752 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.241685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-tmp-dir\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.342859 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.342820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-home\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.342859 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.342862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-dshm\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.343099 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.342903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad0d16e-c06e-4341-8cb4-51908394bf61-tls-certs\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.343099 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.342926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-tmp-dir\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.343099 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.342989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-model-cache\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.343099 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.343021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzx6z\" (UniqueName: \"kubernetes.io/projected/6ad0d16e-c06e-4341-8cb4-51908394bf61-kube-api-access-zzx6z\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.343300 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.343283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-home\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.343414 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.343358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-tmp-dir\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.343458 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.343423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-model-cache\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.345195 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.345175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-dshm\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.345477 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.345457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad0d16e-c06e-4341-8cb4-51908394bf61-tls-certs\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.350036 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.350013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzx6z\" (UniqueName: \"kubernetes.io/projected/6ad0d16e-c06e-4341-8cb4-51908394bf61-kube-api-access-zzx6z\") pod \"sub-test-model-fcd402-kserve-5df5bf4745-6pzgd\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.498177 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.498079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:36.766093 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:36.766069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"]
Sep 17 01:50:37.559106 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:37.559075    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:50:37.559465 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:37.559149    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:50:37.616002 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:37.615961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd" event={"ID":"6ad0d16e-c06e-4341-8cb4-51908394bf61","Type":"ContainerStarted","Data":"a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df"}
Sep 17 01:50:37.616156 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:37.616015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd" event={"ID":"6ad0d16e-c06e-4341-8cb4-51908394bf61","Type":"ContainerStarted","Data":"50335a51d1f6c6af09a84c30d8b0b0b704c1fc8c129f50c8e6e6c08145601b28"}
Sep 17 01:50:37.631878 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:37.631834    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd" podStartSLOduration=1.631819085 podStartE2EDuration="1.631819085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:37.630882806 +0000 UTC m=+1716.301398898" watchObservedRunningTime="2026-09-17 01:50:37.631819085 +0000 UTC m=+1716.302335172"
Sep 17 01:50:40.121519 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.121461    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8" podUID="43f93183-62ab-42e8-9cdc-a72a86515cee" containerName="main" containerID="cri-o://55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7" gracePeriod=45
Sep 17 01:50:40.372243 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.372174    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:50:40.486689 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.486644    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-tmp-dir\") pod \"43f93183-62ab-42e8-9cdc-a72a86515cee\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") "
Sep 17 01:50:40.486885 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.486730    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8njj7\" (UniqueName: \"kubernetes.io/projected/43f93183-62ab-42e8-9cdc-a72a86515cee-kube-api-access-8njj7\") pod \"43f93183-62ab-42e8-9cdc-a72a86515cee\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") "
Sep 17 01:50:40.486885 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.486765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-home\") pod \"43f93183-62ab-42e8-9cdc-a72a86515cee\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") "
Sep 17 01:50:40.486885 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.486820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f93183-62ab-42e8-9cdc-a72a86515cee-tls-certs\") pod \"43f93183-62ab-42e8-9cdc-a72a86515cee\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") "
Sep 17 01:50:40.487055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.486888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-model-cache\") pod \"43f93183-62ab-42e8-9cdc-a72a86515cee\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") "
Sep 17 01:50:40.487055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.486928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-dshm\") pod \"43f93183-62ab-42e8-9cdc-a72a86515cee\" (UID: \"43f93183-62ab-42e8-9cdc-a72a86515cee\") "
Sep 17 01:50:40.487055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.486968    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43f93183-62ab-42e8-9cdc-a72a86515cee" (UID: "43f93183-62ab-42e8-9cdc-a72a86515cee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:50:40.487222 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.487062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-home" (OuterVolumeSpecName: "home") pod "43f93183-62ab-42e8-9cdc-a72a86515cee" (UID: "43f93183-62ab-42e8-9cdc-a72a86515cee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:50:40.487222 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.487148    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-model-cache" (OuterVolumeSpecName: "model-cache") pod "43f93183-62ab-42e8-9cdc-a72a86515cee" (UID: "43f93183-62ab-42e8-9cdc-a72a86515cee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:50:40.487421 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.487403    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:40.487480 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.487426    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:40.487480 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.487440    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:40.489239 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.489207    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-dshm" (OuterVolumeSpecName: "dshm") pod "43f93183-62ab-42e8-9cdc-a72a86515cee" (UID: "43f93183-62ab-42e8-9cdc-a72a86515cee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:50:40.489426 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.489405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f93183-62ab-42e8-9cdc-a72a86515cee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43f93183-62ab-42e8-9cdc-a72a86515cee" (UID: "43f93183-62ab-42e8-9cdc-a72a86515cee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:50:40.489570 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.489526    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f93183-62ab-42e8-9cdc-a72a86515cee-kube-api-access-8njj7" (OuterVolumeSpecName: "kube-api-access-8njj7") pod "43f93183-62ab-42e8-9cdc-a72a86515cee" (UID: "43f93183-62ab-42e8-9cdc-a72a86515cee"). InnerVolumeSpecName "kube-api-access-8njj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:50:40.588262 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.588222    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8njj7\" (UniqueName: \"kubernetes.io/projected/43f93183-62ab-42e8-9cdc-a72a86515cee-kube-api-access-8njj7\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:40.588262 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.588256    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f93183-62ab-42e8-9cdc-a72a86515cee-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:40.588262 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.588266    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f93183-62ab-42e8-9cdc-a72a86515cee-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:40.633748 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.633658    2578 generic.go:356] "Generic (PLEG): container finished" podID="43f93183-62ab-42e8-9cdc-a72a86515cee" containerID="55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7" exitCode=0
Sep 17 01:50:40.633748 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.633704    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"
Sep 17 01:50:40.633922 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.633751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8" event={"ID":"43f93183-62ab-42e8-9cdc-a72a86515cee","Type":"ContainerDied","Data":"55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7"}
Sep 17 01:50:40.633922 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.633797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8" event={"ID":"43f93183-62ab-42e8-9cdc-a72a86515cee","Type":"ContainerDied","Data":"b582471f0111efdc2e06f2a330792b4623d47753e7590e03a0c4c5b06eaf962f"}
Sep 17 01:50:40.633922 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.633816    2578 scope.go:117] "RemoveContainer" containerID="55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7"
Sep 17 01:50:40.660024 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.643634    2578 scope.go:117] "RemoveContainer" containerID="55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7"
Sep 17 01:50:40.660024 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:50:40.643989    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7\": container with ID starting with 55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7 not found: ID does not exist" containerID="55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7"
Sep 17 01:50:40.660024 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.644022    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7"} err="failed to get container status \"55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7\": rpc error: code = NotFound desc = could not find container \"55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7\": container with ID starting with 55084e70e2a0f5112115e8ead13113907c75a60fb06128dfb5550118faf0bce7 not found: ID does not exist"
Sep 17 01:50:40.660024 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.655035    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"]
Sep 17 01:50:40.661012 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:40.660992    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/auth-test-model-fcd402-kserve-5fcc65798c-9knl8"]
Sep 17 01:50:41.640767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:41.640734    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerID="90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274" exitCode=0
Sep 17 01:50:41.641221 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:41.640808    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" event={"ID":"ddd0a697-a92e-4b5d-860a-93836be86cd0","Type":"ContainerDied","Data":"90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274"}
Sep 17 01:50:41.641279 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:41.641267    2578 scope.go:117] "RemoveContainer" containerID="90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274"
Sep 17 01:50:41.642528 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:41.642503    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:50:41.642602 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:41.642589    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:50:41.642638 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:41.642619    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:50:41.921696 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:41.921604    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43f93183-62ab-42e8-9cdc-a72a86515cee" path="/var/lib/kubelet/pods/43f93183-62ab-42e8-9cdc-a72a86515cee/volumes"
Sep 17 01:50:42.649591 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:42.649554    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" event={"ID":"ddd0a697-a92e-4b5d-860a-93836be86cd0","Type":"ContainerStarted","Data":"b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737"}
Sep 17 01:50:42.679233 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:42.679158    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf" podUID="f3c39857-81ae-4d39-a06d-95f4e115951b" containerName="main" containerID="cri-o://f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384" gracePeriod=45
Sep 17 01:50:42.996029 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:42.995996    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:50:43.115910 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.115873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c39857-81ae-4d39-a06d-95f4e115951b-tls-certs\") pod \"f3c39857-81ae-4d39-a06d-95f4e115951b\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") "
Sep 17 01:50:43.116093 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.115965    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-dshm\") pod \"f3c39857-81ae-4d39-a06d-95f4e115951b\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") "
Sep 17 01:50:43.116093 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.116068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28vcs\" (UniqueName: \"kubernetes.io/projected/f3c39857-81ae-4d39-a06d-95f4e115951b-kube-api-access-28vcs\") pod \"f3c39857-81ae-4d39-a06d-95f4e115951b\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") "
Sep 17 01:50:43.116219 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.116102    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-model-cache\") pod \"f3c39857-81ae-4d39-a06d-95f4e115951b\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") "
Sep 17 01:50:43.116279 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.116225    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-home\") pod \"f3c39857-81ae-4d39-a06d-95f4e115951b\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") "
Sep 17 01:50:43.116279 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.116251    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-tmp-dir\") pod \"f3c39857-81ae-4d39-a06d-95f4e115951b\" (UID: \"f3c39857-81ae-4d39-a06d-95f4e115951b\") "
Sep 17 01:50:43.116416 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.116389    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3c39857-81ae-4d39-a06d-95f4e115951b" (UID: "f3c39857-81ae-4d39-a06d-95f4e115951b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:50:43.116481 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.116434    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-home" (OuterVolumeSpecName: "home") pod "f3c39857-81ae-4d39-a06d-95f4e115951b" (UID: "f3c39857-81ae-4d39-a06d-95f4e115951b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:50:43.116606 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.116565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3c39857-81ae-4d39-a06d-95f4e115951b" (UID: "f3c39857-81ae-4d39-a06d-95f4e115951b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:50:43.116713 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.116611    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:43.116713 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.116626    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:43.118212 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.118191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c39857-81ae-4d39-a06d-95f4e115951b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3c39857-81ae-4d39-a06d-95f4e115951b" (UID: "f3c39857-81ae-4d39-a06d-95f4e115951b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:50:43.118299 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.118274    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-dshm" (OuterVolumeSpecName: "dshm") pod "f3c39857-81ae-4d39-a06d-95f4e115951b" (UID: "f3c39857-81ae-4d39-a06d-95f4e115951b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:50:43.118346 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.118295    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c39857-81ae-4d39-a06d-95f4e115951b-kube-api-access-28vcs" (OuterVolumeSpecName: "kube-api-access-28vcs") pod "f3c39857-81ae-4d39-a06d-95f4e115951b" (UID: "f3c39857-81ae-4d39-a06d-95f4e115951b"). InnerVolumeSpecName "kube-api-access-28vcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:50:43.161690 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.161660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:43.161889 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.161867    2578 patch_prober.go:28] interesting pod/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 01:50:43.161937 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.161915    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 01:50:43.217850 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.217768    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:43.217850 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.217795    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c39857-81ae-4d39-a06d-95f4e115951b-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:43.217850 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.217804    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c39857-81ae-4d39-a06d-95f4e115951b-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:43.217850 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.217813    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28vcs\" (UniqueName: \"kubernetes.io/projected/f3c39857-81ae-4d39-a06d-95f4e115951b-kube-api-access-28vcs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:50:43.656207 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.656172    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3c39857-81ae-4d39-a06d-95f4e115951b" containerID="f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384" exitCode=0
Sep 17 01:50:43.656647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.656213    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"
Sep 17 01:50:43.656647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.656252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf" event={"ID":"f3c39857-81ae-4d39-a06d-95f4e115951b","Type":"ContainerDied","Data":"f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384"}
Sep 17 01:50:43.656647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.656294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf" event={"ID":"f3c39857-81ae-4d39-a06d-95f4e115951b","Type":"ContainerDied","Data":"ee5a46e16d8050059de92407fb44e0f8d9b02847d87394619af1bc8ae8b6ce99"}
Sep 17 01:50:43.656647 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.656312    2578 scope.go:117] "RemoveContainer" containerID="f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384"
Sep 17 01:50:43.665577 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.665554    2578 scope.go:117] "RemoveContainer" containerID="f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384"
Sep 17 01:50:43.665860 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:50:43.665843    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384\": container with ID starting with f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384 not found: ID does not exist" containerID="f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384"
Sep 17 01:50:43.665924 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.665868    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384"} err="failed to get container status \"f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384\": rpc error: code = NotFound desc = could not find container \"f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384\": container with ID starting with f84ec064da5f552185a4e4572f8e41cb62f8d79ae11caabc3f34a50bbb3b5384 not found: ID does not exist"
Sep 17 01:50:43.677970 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.677942    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"]
Sep 17 01:50:43.681395 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.681363    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/auth-test-model-361849-kserve-7d4cc4c487-m7bhf"]
Sep 17 01:50:43.920597 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:43.920513    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c39857-81ae-4d39-a06d-95f4e115951b" path="/var/lib/kubelet/pods/f3c39857-81ae-4d39-a06d-95f4e115951b/volumes"
Sep 17 01:50:44.162403 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:44.162360    2578 patch_prober.go:28] interesting pod/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 01:50:44.162634 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:44.162430    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 01:50:45.162805 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:45.162771    2578 patch_prober.go:28] interesting pod/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 01:50:45.163205 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:45.162830    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 01:50:46.162831 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:46.162796    2578 patch_prober.go:28] interesting pod/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 01:50:46.163217 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:46.162853    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 01:50:46.499119 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:46.499009    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:46.499119 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:46.499106    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:46.511951 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:46.511920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:46.704345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:46.704305    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:50:47.161854 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:47.161812    2578 patch_prober.go:28] interesting pod/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 01:50:47.162062 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:47.161873    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 01:50:48.008421 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.008386    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"]
Sep 17 01:50:48.008989 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.008969    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f93183-62ab-42e8-9cdc-a72a86515cee" containerName="main"
Sep 17 01:50:48.009038 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.008993    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f93183-62ab-42e8-9cdc-a72a86515cee" containerName="main"
Sep 17 01:50:48.009038 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.009023    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c39857-81ae-4d39-a06d-95f4e115951b" containerName="main"
Sep 17 01:50:48.009038 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.009030    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c39857-81ae-4d39-a06d-95f4e115951b" containerName="main"
Sep 17 01:50:48.009135 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.009116    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f93183-62ab-42e8-9cdc-a72a86515cee" containerName="main"
Sep 17 01:50:48.009135 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.009130    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c39857-81ae-4d39-a06d-95f4e115951b" containerName="main"
Sep 17 01:50:48.014871 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.014850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.017494 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.017472    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6f3c6f-openshift-default-dockercfg-6fjjx\""
Sep 17 01:50:48.031642 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.031612    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"]
Sep 17 01:50:48.065388 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.065327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-envoy\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.065633 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.065445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-podinfo\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.065633 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.065481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-socket\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.065633 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.065505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istiod-ca-cert\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.065633 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.065602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-credential-socket\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.065873 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.065674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-data\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.065873 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.065723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-token\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.065873 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.065743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxhd\" (UniqueName: \"kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-kube-api-access-kqxhd\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.065873 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.065780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-certs\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.166477 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.166447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:48.166733 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.166711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-certs\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.166840 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.166753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-envoy\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.166840 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.166835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-podinfo\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.166954 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.166866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-socket\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.166954 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.166890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istiod-ca-cert\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.166954 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.166900    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:48.167110 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.166958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-credential-socket\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.167110 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.167015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-data\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.167110 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.167050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-token\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.167110 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.167077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxhd\" (UniqueName: \"kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-kube-api-access-kqxhd\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.167315 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.167150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-certs\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.167610 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.167581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-socket\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.167843 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.167825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-data\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.167965 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.167837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istiod-ca-cert\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.167965 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.167865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-credential-socket\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.168159 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.168014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:50:48.169503 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.169480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-podinfo\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.169648 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.169561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-envoy\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.175090 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.175070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxhd\" (UniqueName: \"kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-kube-api-access-kqxhd\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.175160 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.175115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-token\") pod \"e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") " pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.325927 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.325895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:48.602216 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.602163    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"]
Sep 17 01:50:48.604508 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:50:48.604477    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab276a1_7f0a_4be7_a0cb_06666d05dab6.slice/crio-9581307066a8e82f5fcd8c4c083af33c948b67310c6a4376932a4a59ad86e89e WatchSource:0}: Error finding container 9581307066a8e82f5fcd8c4c083af33c948b67310c6a4376932a4a59ad86e89e: Status 404 returned error can't find the container with id 9581307066a8e82f5fcd8c4c083af33c948b67310c6a4376932a4a59ad86e89e
Sep 17 01:50:48.607233 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.607194    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:50:48.607329 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.607282    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:50:48.607329 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.607322    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:50:48.696731 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.696696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" event={"ID":"4ab276a1-7f0a-4be7-a0cb-06666d05dab6","Type":"ContainerStarted","Data":"7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643"}
Sep 17 01:50:48.696899 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.696751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" event={"ID":"4ab276a1-7f0a-4be7-a0cb-06666d05dab6","Type":"ContainerStarted","Data":"9581307066a8e82f5fcd8c4c083af33c948b67310c6a4376932a4a59ad86e89e"}
Sep 17 01:50:48.715810 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:48.715756    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" podStartSLOduration=1.7157392169999999 podStartE2EDuration="1.715739217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:48.71409615 +0000 UTC m=+1727.384612239" watchObservedRunningTime="2026-09-17 01:50:48.715739217 +0000 UTC m=+1727.386255305"
Sep 17 01:50:49.326957 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:49.326919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:49.332162 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:49.332131    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:49.703132 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:49.703037    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:49.703790 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:49.703772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:50:54.937799 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:54.937764    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"]
Sep 17 01:50:54.942178 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:54.942157    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:50:54.944711 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:54.944679    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6f3c6f\""
Sep 17 01:50:54.944823 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:54.944734    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6f3c6f-dockercfg-8cnhq\""
Sep 17 01:50:54.952010 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:54.951987    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"]
Sep 17 01:50:54.987882 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:54.987853    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"]
Sep 17 01:50:54.991709 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:54.991693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:50:55.000670 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.000645    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"]
Sep 17 01:50:55.034047 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.034019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85h7\" (UniqueName: \"kubernetes.io/projected/29bc7957-1077-4bec-a369-c99af84e5468-kube-api-access-s85h7\") pod \"payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9\" (UID: \"29bc7957-1077-4bec-a369-c99af84e5468\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:50:55.034175 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.034057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bc7957-1077-4bec-a369-c99af84e5468-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9\" (UID: \"29bc7957-1077-4bec-a369-c99af84e5468\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:50:55.135393 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.135356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56de9ca-1bc5-4053-b2c1-d8a79935f588-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8\" (UID: \"f56de9ca-1bc5-4053-b2c1-d8a79935f588\") " pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:50:55.135614 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.135454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8sms\" (UniqueName: \"kubernetes.io/projected/f56de9ca-1bc5-4053-b2c1-d8a79935f588-kube-api-access-r8sms\") pod \"payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8\" (UID: \"f56de9ca-1bc5-4053-b2c1-d8a79935f588\") " pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:50:55.135614 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.135480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85h7\" (UniqueName: \"kubernetes.io/projected/29bc7957-1077-4bec-a369-c99af84e5468-kube-api-access-s85h7\") pod \"payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9\" (UID: \"29bc7957-1077-4bec-a369-c99af84e5468\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:50:55.135614 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.135515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bc7957-1077-4bec-a369-c99af84e5468-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9\" (UID: \"29bc7957-1077-4bec-a369-c99af84e5468\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:50:55.136299 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.136280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bc7957-1077-4bec-a369-c99af84e5468-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9\" (UID: \"29bc7957-1077-4bec-a369-c99af84e5468\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:50:55.143121 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.143095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85h7\" (UniqueName: \"kubernetes.io/projected/29bc7957-1077-4bec-a369-c99af84e5468-kube-api-access-s85h7\") pod \"payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9\" (UID: \"29bc7957-1077-4bec-a369-c99af84e5468\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:50:55.236391 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.236317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8sms\" (UniqueName: \"kubernetes.io/projected/f56de9ca-1bc5-4053-b2c1-d8a79935f588-kube-api-access-r8sms\") pod \"payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8\" (UID: \"f56de9ca-1bc5-4053-b2c1-d8a79935f588\") " pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:50:55.236548 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.236413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56de9ca-1bc5-4053-b2c1-d8a79935f588-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8\" (UID: \"f56de9ca-1bc5-4053-b2c1-d8a79935f588\") " pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:50:55.237114 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.237096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56de9ca-1bc5-4053-b2c1-d8a79935f588-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8\" (UID: \"f56de9ca-1bc5-4053-b2c1-d8a79935f588\") " pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:50:55.244118 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.244097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8sms\" (UniqueName: \"kubernetes.io/projected/f56de9ca-1bc5-4053-b2c1-d8a79935f588-kube-api-access-r8sms\") pod \"payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8\" (UID: \"f56de9ca-1bc5-4053-b2c1-d8a79935f588\") " pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:50:55.253953 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.253931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:50:55.303888 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.303671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:50:55.523272 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.523248    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"]
Sep 17 01:50:55.525000 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:50:55.524979    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bc7957_1077_4bec_a369_c99af84e5468.slice/crio-86c9407f56ef5de5f0a5e714197f75baa58a7967f30ee8450f669d04e3d21619 WatchSource:0}: Error finding container 86c9407f56ef5de5f0a5e714197f75baa58a7967f30ee8450f669d04e3d21619: Status 404 returned error can't find the container with id 86c9407f56ef5de5f0a5e714197f75baa58a7967f30ee8450f669d04e3d21619
Sep 17 01:50:55.584063 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.584042    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"]
Sep 17 01:50:55.585641 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:50:55.585616    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56de9ca_1bc5_4053_b2c1_d8a79935f588.slice/crio-eef6d9e066614c57567e1398107fcc09428fdc1c1147184759381138b9a9bbc5 WatchSource:0}: Error finding container eef6d9e066614c57567e1398107fcc09428fdc1c1147184759381138b9a9bbc5: Status 404 returned error can't find the container with id eef6d9e066614c57567e1398107fcc09428fdc1c1147184759381138b9a9bbc5
Sep 17 01:50:55.736236 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.736200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8" event={"ID":"f56de9ca-1bc5-4053-b2c1-d8a79935f588","Type":"ContainerStarted","Data":"3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80"}
Sep 17 01:50:55.736435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.736257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8" event={"ID":"f56de9ca-1bc5-4053-b2c1-d8a79935f588","Type":"ContainerStarted","Data":"eef6d9e066614c57567e1398107fcc09428fdc1c1147184759381138b9a9bbc5"}
Sep 17 01:50:55.736435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.736284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:50:55.737960 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.737934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9" event={"ID":"29bc7957-1077-4bec-a369-c99af84e5468","Type":"ContainerStarted","Data":"67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e"}
Sep 17 01:50:55.738067 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.737975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9" event={"ID":"29bc7957-1077-4bec-a369-c99af84e5468","Type":"ContainerStarted","Data":"86c9407f56ef5de5f0a5e714197f75baa58a7967f30ee8450f669d04e3d21619"}
Sep 17 01:50:55.738067 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.738035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:50:55.755102 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.755049    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8" podStartSLOduration=1.755029078 podStartE2EDuration="1.755029078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:55.751632342 +0000 UTC m=+1734.422148434" watchObservedRunningTime="2026-09-17 01:50:55.755029078 +0000 UTC m=+1734.425545167"
Sep 17 01:50:55.767727 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:50:55.767677    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9" podStartSLOduration=1.7676582299999999 podStartE2EDuration="1.76765823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:55.766664765 +0000 UTC m=+1734.437180854" watchObservedRunningTime="2026-09-17 01:50:55.76765823 +0000 UTC m=+1734.438174322"
Sep 17 01:51:06.745456 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:06.745423    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:51:06.745950 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:06.745754    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:51:10.828788 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:10.828753    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerID="7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643" exitCode=0
Sep 17 01:51:10.829260 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:10.828822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" event={"ID":"4ab276a1-7f0a-4be7-a0cb-06666d05dab6","Type":"ContainerDied","Data":"7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643"}
Sep 17 01:51:10.829260 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:10.829235    2578 scope.go:117] "RemoveContainer" containerID="7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643"
Sep 17 01:51:10.830368 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:10.830344    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:51:10.830417 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:10.830409    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:51:10.830452 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:10.830440    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:51:11.836058 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:11.836025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" event={"ID":"4ab276a1-7f0a-4be7-a0cb-06666d05dab6","Type":"ContainerStarted","Data":"2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f"}
Sep 17 01:51:12.326621 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:12.326577    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:51:12.326834 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:12.326813    2578 patch_prober.go:28] interesting pod/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 17 01:51:12.326882 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:12.326862    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 17 01:51:13.326643 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:13.326602    2578 patch_prober.go:28] interesting pod/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 17 01:51:13.327029 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:13.326661    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 17 01:51:14.326961 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:14.326918    2578 patch_prober.go:28] interesting pod/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 17 01:51:14.327432 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:14.326978    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 17 01:51:15.326415 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:15.326379    2578 patch_prober.go:28] interesting pod/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 17 01:51:15.326621 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:15.326436    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 17 01:51:16.326980 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:16.326943    2578 patch_prober.go:28] interesting pod/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 17 01:51:16.327374 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:16.327002    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 17 01:51:17.401891 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:17.401837    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:51:17.402360 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:17.402293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:51:17.403077 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:17.403056    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:51:39.002402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.002367    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"]
Sep 17 01:51:39.003020 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.002660    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7" podUID="d0299d70-4e8a-42be-a620-d1f344ceaaa2" containerName="maas-api" containerID="cri-o://32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2" gracePeriod=30
Sep 17 01:51:39.051969 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.051931    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"]
Sep 17 01:51:39.052264 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.052222    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt" podUID="37a7224c-1a69-4509-888f-46d9a0f77553" containerName="payload-processing" containerID="cri-o://96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5" gracePeriod=30
Sep 17 01:51:39.060863 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.060827    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"]
Sep 17 01:51:39.061127 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.061078    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz" podUID="ec56441a-baee-487c-914d-6bec7d2107c0" containerName="payload-pre-processing" containerID="cri-o://85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff" gracePeriod=30
Sep 17 01:51:39.353665 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.353638    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:51:39.356875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.356855    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:51:39.360455 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.360438    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:51:39.488098 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0299d70-4e8a-42be-a620-d1f344ceaaa2-openshift-service-ca\") pod \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") "
Sep 17 01:51:39.488308 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r699r\" (UniqueName: \"kubernetes.io/projected/d0299d70-4e8a-42be-a620-d1f344ceaaa2-kube-api-access-r699r\") pod \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") "
Sep 17 01:51:39.488308 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488163    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a7224c-1a69-4509-888f-46d9a0f77553-plugins-config-volume\") pod \"37a7224c-1a69-4509-888f-46d9a0f77553\" (UID: \"37a7224c-1a69-4509-888f-46d9a0f77553\") "
Sep 17 01:51:39.488308 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488270    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-metrics-certs\") pod \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") "
Sep 17 01:51:39.488493 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488322    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec56441a-baee-487c-914d-6bec7d2107c0-plugins-config-volume\") pod \"ec56441a-baee-487c-914d-6bec7d2107c0\" (UID: \"ec56441a-baee-487c-914d-6bec7d2107c0\") "
Sep 17 01:51:39.488493 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488366    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f59dx\" (UniqueName: \"kubernetes.io/projected/37a7224c-1a69-4509-888f-46d9a0f77553-kube-api-access-f59dx\") pod \"37a7224c-1a69-4509-888f-46d9a0f77553\" (UID: \"37a7224c-1a69-4509-888f-46d9a0f77553\") "
Sep 17 01:51:39.488493 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488482    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tc24\" (UniqueName: \"kubernetes.io/projected/ec56441a-baee-487c-914d-6bec7d2107c0-kube-api-access-6tc24\") pod \"ec56441a-baee-487c-914d-6bec7d2107c0\" (UID: \"ec56441a-baee-487c-914d-6bec7d2107c0\") "
Sep 17 01:51:39.488694 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488499    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0299d70-4e8a-42be-a620-d1f344ceaaa2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0299d70-4e8a-42be-a620-d1f344ceaaa2" (UID: "d0299d70-4e8a-42be-a620-d1f344ceaaa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:51:39.488694 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488513    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a7224c-1a69-4509-888f-46d9a0f77553-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37a7224c-1a69-4509-888f-46d9a0f77553" (UID: "37a7224c-1a69-4509-888f-46d9a0f77553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:51:39.488694 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488523    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-maas-api-tls\") pod \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\" (UID: \"d0299d70-4e8a-42be-a620-d1f344ceaaa2\") "
Sep 17 01:51:39.488853 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.488796    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec56441a-baee-487c-914d-6bec7d2107c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec56441a-baee-487c-914d-6bec7d2107c0" (UID: "ec56441a-baee-487c-914d-6bec7d2107c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:51:39.489122 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.489099    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a7224c-1a69-4509-888f-46d9a0f77553-plugins-config-volume\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:39.489228 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.489128    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec56441a-baee-487c-914d-6bec7d2107c0-plugins-config-volume\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:39.489228 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.489143    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0299d70-4e8a-42be-a620-d1f344ceaaa2-openshift-service-ca\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:39.490546 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.490507    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0299d70-4e8a-42be-a620-d1f344ceaaa2" (UID: "d0299d70-4e8a-42be-a620-d1f344ceaaa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:51:39.490649 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.490568    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0299d70-4e8a-42be-a620-d1f344ceaaa2-kube-api-access-r699r" (OuterVolumeSpecName: "kube-api-access-r699r") pod "d0299d70-4e8a-42be-a620-d1f344ceaaa2" (UID: "d0299d70-4e8a-42be-a620-d1f344ceaaa2"). InnerVolumeSpecName "kube-api-access-r699r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:51:39.490649 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.490602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a7224c-1a69-4509-888f-46d9a0f77553-kube-api-access-f59dx" (OuterVolumeSpecName: "kube-api-access-f59dx") pod "37a7224c-1a69-4509-888f-46d9a0f77553" (UID: "37a7224c-1a69-4509-888f-46d9a0f77553"). InnerVolumeSpecName "kube-api-access-f59dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:51:39.490767 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.490726    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d0299d70-4e8a-42be-a620-d1f344ceaaa2" (UID: "d0299d70-4e8a-42be-a620-d1f344ceaaa2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:51:39.490938 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.490919    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec56441a-baee-487c-914d-6bec7d2107c0-kube-api-access-6tc24" (OuterVolumeSpecName: "kube-api-access-6tc24") pod "ec56441a-baee-487c-914d-6bec7d2107c0" (UID: "ec56441a-baee-487c-914d-6bec7d2107c0"). InnerVolumeSpecName "kube-api-access-6tc24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:51:39.590746 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.590708    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-metrics-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:39.590746 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.590741    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59dx\" (UniqueName: \"kubernetes.io/projected/37a7224c-1a69-4509-888f-46d9a0f77553-kube-api-access-f59dx\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:39.590746 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.590752    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tc24\" (UniqueName: \"kubernetes.io/projected/ec56441a-baee-487c-914d-6bec7d2107c0-kube-api-access-6tc24\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:39.590987 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.590760    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0299d70-4e8a-42be-a620-d1f344ceaaa2-maas-api-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:39.590987 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.590768    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r699r\" (UniqueName: \"kubernetes.io/projected/d0299d70-4e8a-42be-a620-d1f344ceaaa2-kube-api-access-r699r\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:39.979273 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.979240    2578 generic.go:356] "Generic (PLEG): container finished" podID="37a7224c-1a69-4509-888f-46d9a0f77553" containerID="96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5" exitCode=0
Sep 17 01:51:39.979464 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.979293    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"
Sep 17 01:51:39.979464 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.979296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt" event={"ID":"37a7224c-1a69-4509-888f-46d9a0f77553","Type":"ContainerDied","Data":"96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5"}
Sep 17 01:51:39.979464 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.979362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt" event={"ID":"37a7224c-1a69-4509-888f-46d9a0f77553","Type":"ContainerDied","Data":"2f882957509acdee6dfa2f34e1758610a33c9e5381731d02c7604863e3403f73"}
Sep 17 01:51:39.979464 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.979394    2578 scope.go:117] "RemoveContainer" containerID="96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5"
Sep 17 01:51:39.981066 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.981031    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz" event={"ID":"ec56441a-baee-487c-914d-6bec7d2107c0","Type":"ContainerDied","Data":"85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff"}
Sep 17 01:51:39.981185 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.981038    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"
Sep 17 01:51:39.981185 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.980926    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec56441a-baee-487c-914d-6bec7d2107c0" containerID="85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff" exitCode=0
Sep 17 01:51:39.981310 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.981237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz" event={"ID":"ec56441a-baee-487c-914d-6bec7d2107c0","Type":"ContainerDied","Data":"2d7a96b4bf5a8d48834c946fb0fdce649d40a547adfede46eaedcdb9ec47e8c7"}
Sep 17 01:51:39.982875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.982843    2578 generic.go:356] "Generic (PLEG): container finished" podID="d0299d70-4e8a-42be-a620-d1f344ceaaa2" containerID="32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2" exitCode=0
Sep 17 01:51:39.982983 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.982886    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"
Sep 17 01:51:39.982983 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.982899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7" event={"ID":"d0299d70-4e8a-42be-a620-d1f344ceaaa2","Type":"ContainerDied","Data":"32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2"}
Sep 17 01:51:39.982983 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.982944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7" event={"ID":"d0299d70-4e8a-42be-a620-d1f344ceaaa2","Type":"ContainerDied","Data":"1790e4f1926707d19c6507b84b8851c9799bb3a37f14a630c1b78fe18d85ce82"}
Sep 17 01:51:39.990194 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.990172    2578 scope.go:117] "RemoveContainer" containerID="96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5"
Sep 17 01:51:39.990647 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:51:39.990616    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5\": container with ID starting with 96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5 not found: ID does not exist" containerID="96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5"
Sep 17 01:51:39.990756 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.990657    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5"} err="failed to get container status \"96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5\": rpc error: code = NotFound desc = could not find container \"96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5\": container with ID starting with 96699957e0a5ad461a9d51a1b1da8fac0d4d5c77411d4eed0e9517b74bff72f5 not found: ID does not exist"
Sep 17 01:51:39.990756 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.990679    2578 scope.go:117] "RemoveContainer" containerID="85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff"
Sep 17 01:51:39.999426 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.999406    2578 scope.go:117] "RemoveContainer" containerID="85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff"
Sep 17 01:51:39.999766 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:51:39.999744    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff\": container with ID starting with 85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff not found: ID does not exist" containerID="85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff"
Sep 17 01:51:39.999869 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.999772    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff"} err="failed to get container status \"85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff\": rpc error: code = NotFound desc = could not find container \"85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff\": container with ID starting with 85df2950acacbba61d354ccdbb5fa9035e68d5c8cb91851a955a2e882e2d7dff not found: ID does not exist"
Sep 17 01:51:39.999869 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:39.999792    2578 scope.go:117] "RemoveContainer" containerID="32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2"
Sep 17 01:51:40.001795 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:40.001762    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"]
Sep 17 01:51:40.006955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:40.006932    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cf28c-6bdf8997-zbcvt"]
Sep 17 01:51:40.009109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:40.009031    2578 scope.go:117] "RemoveContainer" containerID="32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2"
Sep 17 01:51:40.009392 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:51:40.009373    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2\": container with ID starting with 32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2 not found: ID does not exist" containerID="32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2"
Sep 17 01:51:40.009466 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:40.009402    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2"} err="failed to get container status \"32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2\": rpc error: code = NotFound desc = could not find container \"32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2\": container with ID starting with 32d08fcc7a4b91deead8a01b44a7dfc432311396aaa2694d61e85e61ea829cd2 not found: ID does not exist"
Sep 17 01:51:40.017198 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:40.017169    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"]
Sep 17 01:51:40.022069 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:40.022042    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cf28c-f9cfc8f-n9fc7"]
Sep 17 01:51:40.030597 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:40.030570    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"]
Sep 17 01:51:40.033889 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:40.033854    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cf28c-86b8bf6db-f8gxz"]
Sep 17 01:51:41.920177 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:41.920136    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37a7224c-1a69-4509-888f-46d9a0f77553" path="/var/lib/kubelet/pods/37a7224c-1a69-4509-888f-46d9a0f77553/volumes"
Sep 17 01:51:41.920562 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:41.920547    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0299d70-4e8a-42be-a620-d1f344ceaaa2" path="/var/lib/kubelet/pods/d0299d70-4e8a-42be-a620-d1f344ceaaa2/volumes"
Sep 17 01:51:41.920877 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:41.920865    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec56441a-baee-487c-914d-6bec7d2107c0" path="/var/lib/kubelet/pods/ec56441a-baee-487c-914d-6bec7d2107c0/volumes"
Sep 17 01:51:45.758830 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:45.758791    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"]
Sep 17 01:51:45.759414 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:45.759081    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy" containerID="cri-o://b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737" gracePeriod=30
Sep 17 01:51:48.169618 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:48.169581    2578 patch_prober.go:28] interesting pod/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 01:51:48.169986 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:48.169662    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 01:51:51.004262 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.004235    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:51:51.042890 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.042802    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerID="b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737" exitCode=0
Sep 17 01:51:51.042890 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.042850    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"
Sep 17 01:51:51.042890 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.042864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" event={"ID":"ddd0a697-a92e-4b5d-860a-93836be86cd0","Type":"ContainerDied","Data":"b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737"}
Sep 17 01:51:51.043158 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.042917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n" event={"ID":"ddd0a697-a92e-4b5d-860a-93836be86cd0","Type":"ContainerDied","Data":"43959e60c8b935810cda01f80adf816c9bdae4665f20b553bfbde72e652d2354"}
Sep 17 01:51:51.043158 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.042947    2578 scope.go:117] "RemoveContainer" containerID="b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737"
Sep 17 01:51:51.052132 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.052112    2578 scope.go:117] "RemoveContainer" containerID="90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274"
Sep 17 01:51:51.060315 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.060297    2578 scope.go:117] "RemoveContainer" containerID="b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737"
Sep 17 01:51:51.060652 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:51:51.060632    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737\": container with ID starting with b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737 not found: ID does not exist" containerID="b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737"
Sep 17 01:51:51.060736 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.060662    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737"} err="failed to get container status \"b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737\": rpc error: code = NotFound desc = could not find container \"b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737\": container with ID starting with b6a9f45734a2965b6002ca32aa2d95684975c01e451ac46b8787015199b8c737 not found: ID does not exist"
Sep 17 01:51:51.060736 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.060682    2578 scope.go:117] "RemoveContainer" containerID="90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274"
Sep 17 01:51:51.060923 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:51:51.060908    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274\": container with ID starting with 90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274 not found: ID does not exist" containerID="90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274"
Sep 17 01:51:51.060971 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.060927    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274"} err="failed to get container status \"90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274\": rpc error: code = NotFound desc = could not find container \"90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274\": container with ID starting with 90dd41e1b368ed20dcf162de88c8f28e6e1bf1c54dd264c00e14419211a05274 not found: ID does not exist"
Sep 17 01:51:51.099089 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099054    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-envoy\") pod \"ddd0a697-a92e-4b5d-860a-93836be86cd0\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") "
Sep 17 01:51:51.099254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd0a697-a92e-4b5d-860a-93836be86cd0-istiod-ca-cert\") pod \"ddd0a697-a92e-4b5d-860a-93836be86cd0\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") "
Sep 17 01:51:51.099254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099129    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-token\") pod \"ddd0a697-a92e-4b5d-860a-93836be86cd0\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") "
Sep 17 01:51:51.099254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-credential-socket\") pod \"ddd0a697-a92e-4b5d-860a-93836be86cd0\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") "
Sep 17 01:51:51.099254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-data\") pod \"ddd0a697-a92e-4b5d-860a-93836be86cd0\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") "
Sep 17 01:51:51.099254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099228    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-certs\") pod \"ddd0a697-a92e-4b5d-860a-93836be86cd0\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") "
Sep 17 01:51:51.099528 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099303    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-podinfo\") pod \"ddd0a697-a92e-4b5d-860a-93836be86cd0\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") "
Sep 17 01:51:51.099528 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099353    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7hpl\" (UniqueName: \"kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-kube-api-access-x7hpl\") pod \"ddd0a697-a92e-4b5d-860a-93836be86cd0\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") "
Sep 17 01:51:51.099528 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099395    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-socket\") pod \"ddd0a697-a92e-4b5d-860a-93836be86cd0\" (UID: \"ddd0a697-a92e-4b5d-860a-93836be86cd0\") "
Sep 17 01:51:51.099528 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099467    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ddd0a697-a92e-4b5d-860a-93836be86cd0" (UID: "ddd0a697-a92e-4b5d-860a-93836be86cd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:51:51.099789 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099653    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd0a697-a92e-4b5d-860a-93836be86cd0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ddd0a697-a92e-4b5d-860a-93836be86cd0" (UID: "ddd0a697-a92e-4b5d-860a-93836be86cd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:51:51.099789 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ddd0a697-a92e-4b5d-860a-93836be86cd0" (UID: "ddd0a697-a92e-4b5d-860a-93836be86cd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:51:51.099789 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099687    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ddd0a697-a92e-4b5d-860a-93836be86cd0" (UID: "ddd0a697-a92e-4b5d-860a-93836be86cd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:51:51.099789 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099716    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-data" (OuterVolumeSpecName: "istio-data") pod "ddd0a697-a92e-4b5d-860a-93836be86cd0" (UID: "ddd0a697-a92e-4b5d-860a-93836be86cd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:51:51.100062 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099802    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-data\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:51.100062 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099817    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:51.100062 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099830    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-workload-socket\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:51.100062 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099842    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd0a697-a92e-4b5d-860a-93836be86cd0-istiod-ca-cert\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:51.100062 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.099856    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-credential-socket\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:51.101519 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.101488    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-token" (OuterVolumeSpecName: "istio-token") pod "ddd0a697-a92e-4b5d-860a-93836be86cd0" (UID: "ddd0a697-a92e-4b5d-860a-93836be86cd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:51:51.101630 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.101598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-kube-api-access-x7hpl" (OuterVolumeSpecName: "kube-api-access-x7hpl") pod "ddd0a697-a92e-4b5d-860a-93836be86cd0" (UID: "ddd0a697-a92e-4b5d-860a-93836be86cd0"). InnerVolumeSpecName "kube-api-access-x7hpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:51:51.101751 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.101717    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ddd0a697-a92e-4b5d-860a-93836be86cd0" (UID: "ddd0a697-a92e-4b5d-860a-93836be86cd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:51:51.101816 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.101731    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ddd0a697-a92e-4b5d-860a-93836be86cd0" (UID: "ddd0a697-a92e-4b5d-860a-93836be86cd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 01:51:51.200922 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.200884    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7hpl\" (UniqueName: \"kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-kube-api-access-x7hpl\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:51.200922 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.200915    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-envoy\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:51.200922 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.200924    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-token\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:51.201164 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.200934    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd0a697-a92e-4b5d-860a-93836be86cd0-istio-podinfo\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:51:51.364242 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.364207    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"]
Sep 17 01:51:51.369921 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.369885    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1cf28c-openshift-default-6d4c6fc7cd-4b58n"]
Sep 17 01:51:51.920017 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:51.919983    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" path="/var/lib/kubelet/pods/ddd0a697-a92e-4b5d-860a-93836be86cd0/volumes"
Sep 17 01:51:57.614709 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.614622    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10047-313389"]
Sep 17 01:51:57.615083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.614994    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy"
Sep 17 01:51:57.615083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615005    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy"
Sep 17 01:51:57.615083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615016    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a7224c-1a69-4509-888f-46d9a0f77553" containerName="payload-processing"
Sep 17 01:51:57.615083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615023    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a7224c-1a69-4509-888f-46d9a0f77553" containerName="payload-processing"
Sep 17 01:51:57.615083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615042    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec56441a-baee-487c-914d-6bec7d2107c0" containerName="payload-pre-processing"
Sep 17 01:51:57.615083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615047    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec56441a-baee-487c-914d-6bec7d2107c0" containerName="payload-pre-processing"
Sep 17 01:51:57.615083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615054    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0299d70-4e8a-42be-a620-d1f344ceaaa2" containerName="maas-api"
Sep 17 01:51:57.615083 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615060    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0299d70-4e8a-42be-a620-d1f344ceaaa2" containerName="maas-api"
Sep 17 01:51:57.615351 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615117    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec56441a-baee-487c-914d-6bec7d2107c0" containerName="payload-pre-processing"
Sep 17 01:51:57.615351 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615129    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37a7224c-1a69-4509-888f-46d9a0f77553" containerName="payload-processing"
Sep 17 01:51:57.615351 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615134    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0299d70-4e8a-42be-a620-d1f344ceaaa2" containerName="maas-api"
Sep 17 01:51:57.615351 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615140    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy"
Sep 17 01:51:57.615351 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.615147    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy"
Sep 17 01:51:57.620018 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.619999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-313389"
Sep 17 01:51:57.623475 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.622524    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qszjr\""
Sep 17 01:51:57.623475 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.623209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10047-313389"]
Sep 17 01:51:57.670926 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.670883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx75v\" (UniqueName: \"kubernetes.io/projected/b37ca089-7563-489e-b7f6-75ad152365e4-kube-api-access-wx75v\") pod \"test-curl-10047-313389\" (UID: \"b37ca089-7563-489e-b7f6-75ad152365e4\") " pod="opendatahub/test-curl-10047-313389"
Sep 17 01:51:57.772394 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.772349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx75v\" (UniqueName: \"kubernetes.io/projected/b37ca089-7563-489e-b7f6-75ad152365e4-kube-api-access-wx75v\") pod \"test-curl-10047-313389\" (UID: \"b37ca089-7563-489e-b7f6-75ad152365e4\") " pod="opendatahub/test-curl-10047-313389"
Sep 17 01:51:57.780040 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.780012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx75v\" (UniqueName: \"kubernetes.io/projected/b37ca089-7563-489e-b7f6-75ad152365e4-kube-api-access-wx75v\") pod \"test-curl-10047-313389\" (UID: \"b37ca089-7563-489e-b7f6-75ad152365e4\") " pod="opendatahub/test-curl-10047-313389"
Sep 17 01:51:57.932180 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:57.932101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-313389"
Sep 17 01:51:58.200189 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:58.200167    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10047-313389"]
Sep 17 01:51:58.202001 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:51:58.201969    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb37ca089_7563_489e_b7f6_75ad152365e4.slice/crio-e7afc15810a9462b21e0949afa63253c929f7fa492200b150c81e0c5b69b6e75 WatchSource:0}: Error finding container e7afc15810a9462b21e0949afa63253c929f7fa492200b150c81e0c5b69b6e75: Status 404 returned error can't find the container with id e7afc15810a9462b21e0949afa63253c929f7fa492200b150c81e0c5b69b6e75
Sep 17 01:51:58.203759 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:58.203738    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 01:51:59.085327 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:51:59.085291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10047-313389" event={"ID":"b37ca089-7563-489e-b7f6-75ad152365e4","Type":"ContainerStarted","Data":"e7afc15810a9462b21e0949afa63253c929f7fa492200b150c81e0c5b69b6e75"}
Sep 17 01:52:01.097072 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.097038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10047-313389" event={"ID":"b37ca089-7563-489e-b7f6-75ad152365e4","Type":"ContainerStarted","Data":"92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b"}
Sep 17 01:52:01.111287 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.111236    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10047-313389" podStartSLOduration=1.964992635 podStartE2EDuration="4.111221111s" totalImagesPullingTime="2.146228476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:52:01.109640028 +0000 UTC m=+1799.780156118" watchObservedRunningTime="2026-09-17 01:52:01.111221111 +0000 UTC m=+1799.781737199"
Sep 17 01:52:01.577833 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.577799    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"]
Sep 17 01:52:01.578067 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.578042    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8" podUID="f56de9ca-1bc5-4053-b2c1-d8a79935f588" containerName="payload-processing" containerID="cri-o://3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80" gracePeriod=30
Sep 17 01:52:01.588785 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.588753    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"]
Sep 17 01:52:01.589025 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.588997    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9" podUID="29bc7957-1077-4bec-a369-c99af84e5468" containerName="payload-pre-processing" containerID="cri-o://67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e" gracePeriod=30
Sep 17 01:52:01.678213 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.678178    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10047-313389"]
Sep 17 01:52:01.680628 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.680594    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10047-313389"]
Sep 17 01:52:01.791731 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.791707    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10047-a14ec7"]
Sep 17 01:52:01.792198 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.792177    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37ca089-7563-489e-b7f6-75ad152365e4" containerName="test-curl-10047-313389"
Sep 17 01:52:01.792317 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.792222    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37ca089-7563-489e-b7f6-75ad152365e4" containerName="test-curl-10047-313389"
Sep 17 01:52:01.792317 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.792265    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy"
Sep 17 01:52:01.792317 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.792288    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd0a697-a92e-4b5d-860a-93836be86cd0" containerName="istio-proxy"
Sep 17 01:52:01.792471 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.792397    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37ca089-7563-489e-b7f6-75ad152365e4" containerName="test-curl-10047-313389"
Sep 17 01:52:01.795793 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.795773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-a14ec7"
Sep 17 01:52:01.803501 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.801943    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10047-a14ec7"]
Sep 17 01:52:01.869458 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.869425    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:52:01.872392 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.872371    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:52:01.918211 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:01.918009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsz9\" (UniqueName: \"kubernetes.io/projected/c4fedf30-2aba-447a-9db5-e3c36e73bfd4-kube-api-access-qrsz9\") pod \"test-curl-10047-a14ec7\" (UID: \"c4fedf30-2aba-447a-9db5-e3c36e73bfd4\") " pod="opendatahub/test-curl-10047-a14ec7"
Sep 17 01:52:02.018986 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.018956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56de9ca-1bc5-4053-b2c1-d8a79935f588-plugins-config-volume\") pod \"f56de9ca-1bc5-4053-b2c1-d8a79935f588\" (UID: \"f56de9ca-1bc5-4053-b2c1-d8a79935f588\") "
Sep 17 01:52:02.019130 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.019021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s85h7\" (UniqueName: \"kubernetes.io/projected/29bc7957-1077-4bec-a369-c99af84e5468-kube-api-access-s85h7\") pod \"29bc7957-1077-4bec-a369-c99af84e5468\" (UID: \"29bc7957-1077-4bec-a369-c99af84e5468\") "
Sep 17 01:52:02.019130 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.019077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bc7957-1077-4bec-a369-c99af84e5468-plugins-config-volume\") pod \"29bc7957-1077-4bec-a369-c99af84e5468\" (UID: \"29bc7957-1077-4bec-a369-c99af84e5468\") "
Sep 17 01:52:02.019130 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.019101    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8sms\" (UniqueName: \"kubernetes.io/projected/f56de9ca-1bc5-4053-b2c1-d8a79935f588-kube-api-access-r8sms\") pod \"f56de9ca-1bc5-4053-b2c1-d8a79935f588\" (UID: \"f56de9ca-1bc5-4053-b2c1-d8a79935f588\") "
Sep 17 01:52:02.019300 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.019246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsz9\" (UniqueName: \"kubernetes.io/projected/c4fedf30-2aba-447a-9db5-e3c36e73bfd4-kube-api-access-qrsz9\") pod \"test-curl-10047-a14ec7\" (UID: \"c4fedf30-2aba-447a-9db5-e3c36e73bfd4\") " pod="opendatahub/test-curl-10047-a14ec7"
Sep 17 01:52:02.019488 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.019461    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56de9ca-1bc5-4053-b2c1-d8a79935f588-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f56de9ca-1bc5-4053-b2c1-d8a79935f588" (UID: "f56de9ca-1bc5-4053-b2c1-d8a79935f588"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:52:02.019659 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.019484    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bc7957-1077-4bec-a369-c99af84e5468-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29bc7957-1077-4bec-a369-c99af84e5468" (UID: "29bc7957-1077-4bec-a369-c99af84e5468"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:52:02.021549 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.021507    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56de9ca-1bc5-4053-b2c1-d8a79935f588-kube-api-access-r8sms" (OuterVolumeSpecName: "kube-api-access-r8sms") pod "f56de9ca-1bc5-4053-b2c1-d8a79935f588" (UID: "f56de9ca-1bc5-4053-b2c1-d8a79935f588"). InnerVolumeSpecName "kube-api-access-r8sms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:52:02.021627 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.021593    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bc7957-1077-4bec-a369-c99af84e5468-kube-api-access-s85h7" (OuterVolumeSpecName: "kube-api-access-s85h7") pod "29bc7957-1077-4bec-a369-c99af84e5468" (UID: "29bc7957-1077-4bec-a369-c99af84e5468"). InnerVolumeSpecName "kube-api-access-s85h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:52:02.029116 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.029091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsz9\" (UniqueName: \"kubernetes.io/projected/c4fedf30-2aba-447a-9db5-e3c36e73bfd4-kube-api-access-qrsz9\") pod \"test-curl-10047-a14ec7\" (UID: \"c4fedf30-2aba-447a-9db5-e3c36e73bfd4\") " pod="opendatahub/test-curl-10047-a14ec7"
Sep 17 01:52:02.103078 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.103038    2578 generic.go:356] "Generic (PLEG): container finished" podID="29bc7957-1077-4bec-a369-c99af84e5468" containerID="67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e" exitCode=0
Sep 17 01:52:02.103514 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.103094    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"
Sep 17 01:52:02.103514 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.103129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9" event={"ID":"29bc7957-1077-4bec-a369-c99af84e5468","Type":"ContainerDied","Data":"67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e"}
Sep 17 01:52:02.103514 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.103209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9" event={"ID":"29bc7957-1077-4bec-a369-c99af84e5468","Type":"ContainerDied","Data":"86c9407f56ef5de5f0a5e714197f75baa58a7967f30ee8450f669d04e3d21619"}
Sep 17 01:52:02.103514 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.103244    2578 scope.go:117] "RemoveContainer" containerID="67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e"
Sep 17 01:52:02.104589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.104568    2578 generic.go:356] "Generic (PLEG): container finished" podID="f56de9ca-1bc5-4053-b2c1-d8a79935f588" containerID="3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80" exitCode=0
Sep 17 01:52:02.104702 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.104591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8" event={"ID":"f56de9ca-1bc5-4053-b2c1-d8a79935f588","Type":"ContainerDied","Data":"3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80"}
Sep 17 01:52:02.104702 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.104615    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"
Sep 17 01:52:02.104702 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.104680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8" event={"ID":"f56de9ca-1bc5-4053-b2c1-d8a79935f588","Type":"ContainerDied","Data":"eef6d9e066614c57567e1398107fcc09428fdc1c1147184759381138b9a9bbc5"}
Sep 17 01:52:02.104922 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:52:02.104906    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10047-313389" container="test-curl-10047-313389"
Sep 17 01:52:02.112802 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.112780    2578 scope.go:117] "RemoveContainer" containerID="67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e"
Sep 17 01:52:02.113125 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:52:02.113105    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e\": container with ID starting with 67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e not found: ID does not exist" containerID="67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e"
Sep 17 01:52:02.113125 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.113122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-a14ec7"
Sep 17 01:52:02.113279 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.113136    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e"} err="failed to get container status \"67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e\": rpc error: code = NotFound desc = could not find container \"67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e\": container with ID starting with 67025c6d39249db24092223a7f402e18d62c39316c287b8cb3e4257effe5576e not found: ID does not exist"
Sep 17 01:52:02.113279 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.113159    2578 scope.go:117] "RemoveContainer" containerID="3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80"
Sep 17 01:52:02.120160 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.120132    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bc7957-1077-4bec-a369-c99af84e5468-plugins-config-volume\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:02.120160 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.120160    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8sms\" (UniqueName: \"kubernetes.io/projected/f56de9ca-1bc5-4053-b2c1-d8a79935f588-kube-api-access-r8sms\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:02.120283 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.120170    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f56de9ca-1bc5-4053-b2c1-d8a79935f588-plugins-config-volume\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:02.120283 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.120178    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85h7\" (UniqueName: \"kubernetes.io/projected/29bc7957-1077-4bec-a369-c99af84e5468-kube-api-access-s85h7\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:02.123581 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.123563    2578 scope.go:117] "RemoveContainer" containerID="3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80"
Sep 17 01:52:02.123858 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:52:02.123836    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80\": container with ID starting with 3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80 not found: ID does not exist" containerID="3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80"
Sep 17 01:52:02.123949 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.123863    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80"} err="failed to get container status \"3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80\": rpc error: code = NotFound desc = could not find container \"3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80\": container with ID starting with 3f69df3aad823093db6af5027d858bfbcb07a883ec5db69260132664a6ddaa80 not found: ID does not exist"
Sep 17 01:52:02.128636 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.128615    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"]
Sep 17 01:52:02.132367 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.132346    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6f3c6f-7467bc7dc4-fk5s9"]
Sep 17 01:52:02.143272 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.143242    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"]
Sep 17 01:52:02.146560 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.146509    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6f3c6f-5658f94b9-vhwh8"]
Sep 17 01:52:02.387843 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:02.387815    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10047-a14ec7"]
Sep 17 01:52:02.390262 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:52:02.390239    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4fedf30_2aba_447a_9db5_e3c36e73bfd4.slice/crio-176e3d4d623daa896c538ec0ee7d1640a1304aab8d52176ec9104fe10765da51 WatchSource:0}: Error finding container 176e3d4d623daa896c538ec0ee7d1640a1304aab8d52176ec9104fe10765da51: Status 404 returned error can't find the container with id 176e3d4d623daa896c538ec0ee7d1640a1304aab8d52176ec9104fe10765da51
Sep 17 01:52:03.112125 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:03.112093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10047-a14ec7" event={"ID":"c4fedf30-2aba-447a-9db5-e3c36e73bfd4","Type":"ContainerStarted","Data":"10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c"}
Sep 17 01:52:03.112578 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:03.112135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10047-a14ec7" event={"ID":"c4fedf30-2aba-447a-9db5-e3c36e73bfd4","Type":"ContainerStarted","Data":"176e3d4d623daa896c538ec0ee7d1640a1304aab8d52176ec9104fe10765da51"}
Sep 17 01:52:03.113198 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:03.113175    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10047-313389" podUID="b37ca089-7563-489e-b7f6-75ad152365e4" containerName="test-curl-10047-313389" containerID="cri-o://92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b" gracePeriod=2
Sep 17 01:52:03.126869 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:03.126825    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10047-a14ec7" podStartSLOduration=2.126811068 podStartE2EDuration="2.126811068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:52:03.125526894 +0000 UTC m=+1801.796042990" watchObservedRunningTime="2026-09-17 01:52:03.126811068 +0000 UTC m=+1801.797327157"
Sep 17 01:52:03.669645 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:03.669613    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10047-a14ec7"]
Sep 17 01:52:03.671996 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:03.671969    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10047-a14ec7"]
Sep 17 01:52:03.921254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:03.921223    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29bc7957-1077-4bec-a369-c99af84e5468" path="/var/lib/kubelet/pods/29bc7957-1077-4bec-a369-c99af84e5468/volumes"
Sep 17 01:52:03.921604 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:03.921590    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f56de9ca-1bc5-4053-b2c1-d8a79935f588" path="/var/lib/kubelet/pods/f56de9ca-1bc5-4053-b2c1-d8a79935f588/volumes"
Sep 17 01:52:04.117909 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:52:04.117877    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10047-a14ec7" container="test-curl-10047-a14ec7"
Sep 17 01:52:05.123406 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:05.123367    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10047-a14ec7" podUID="c4fedf30-2aba-447a-9db5-e3c36e73bfd4" containerName="test-curl-10047-a14ec7" containerID="cri-o://10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c" gracePeriod=2
Sep 17 01:52:05.284933 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:05.284909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-313389"
Sep 17 01:52:05.452277 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:05.452190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx75v\" (UniqueName: \"kubernetes.io/projected/b37ca089-7563-489e-b7f6-75ad152365e4-kube-api-access-wx75v\") pod \"b37ca089-7563-489e-b7f6-75ad152365e4\" (UID: \"b37ca089-7563-489e-b7f6-75ad152365e4\") "
Sep 17 01:52:05.454222 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:05.454193    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37ca089-7563-489e-b7f6-75ad152365e4-kube-api-access-wx75v" (OuterVolumeSpecName: "kube-api-access-wx75v") pod "b37ca089-7563-489e-b7f6-75ad152365e4" (UID: "b37ca089-7563-489e-b7f6-75ad152365e4"). InnerVolumeSpecName "kube-api-access-wx75v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:52:05.553403 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:05.553368    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx75v\" (UniqueName: \"kubernetes.io/projected/b37ca089-7563-489e-b7f6-75ad152365e4-kube-api-access-wx75v\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:05.618572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:05.618527    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"]
Sep 17 01:52:05.920950 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:05.920914    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b37ca089-7563-489e-b7f6-75ad152365e4" path="/var/lib/kubelet/pods/b37ca089-7563-489e-b7f6-75ad152365e4/volumes"
Sep 17 01:52:06.129409 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:06.129379    2578 generic.go:356] "Generic (PLEG): container finished" podID="b37ca089-7563-489e-b7f6-75ad152365e4" containerID="92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b" exitCode=137
Sep 17 01:52:06.129409 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:06.129413    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-313389"
Sep 17 01:52:06.129892 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:06.129440    2578 scope.go:117] "RemoveContainer" containerID="92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b"
Sep 17 01:52:06.137614 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:06.137590    2578 scope.go:117] "RemoveContainer" containerID="92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b"
Sep 17 01:52:06.137887 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:52:06.137867    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b\": container with ID starting with 92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b not found: ID does not exist" containerID="92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b"
Sep 17 01:52:06.137944 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:06.137893    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b"} err="failed to get container status \"92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b\": rpc error: code = NotFound desc = could not find container \"92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b\": container with ID starting with 92fbf4139c915eff1bcd6776ee93f6302c73c0dd70c2aeaf4c0d672436d27d7b not found: ID does not exist"
Sep 17 01:52:07.300037 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:07.300010    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-a14ec7"
Sep 17 01:52:07.472349 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:07.472270    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrsz9\" (UniqueName: \"kubernetes.io/projected/c4fedf30-2aba-447a-9db5-e3c36e73bfd4-kube-api-access-qrsz9\") pod \"c4fedf30-2aba-447a-9db5-e3c36e73bfd4\" (UID: \"c4fedf30-2aba-447a-9db5-e3c36e73bfd4\") "
Sep 17 01:52:07.474258 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:07.474235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fedf30-2aba-447a-9db5-e3c36e73bfd4-kube-api-access-qrsz9" (OuterVolumeSpecName: "kube-api-access-qrsz9") pod "c4fedf30-2aba-447a-9db5-e3c36e73bfd4" (UID: "c4fedf30-2aba-447a-9db5-e3c36e73bfd4"). InnerVolumeSpecName "kube-api-access-qrsz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:52:07.573906 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:07.573875    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrsz9\" (UniqueName: \"kubernetes.io/projected/c4fedf30-2aba-447a-9db5-e3c36e73bfd4-kube-api-access-qrsz9\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:07.920823 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:07.920790    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4fedf30-2aba-447a-9db5-e3c36e73bfd4" path="/var/lib/kubelet/pods/c4fedf30-2aba-447a-9db5-e3c36e73bfd4/volumes"
Sep 17 01:52:08.094325 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:08.094291    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"]
Sep 17 01:52:08.094766 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:08.094735    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy" containerID="cri-o://2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f" gracePeriod=30
Sep 17 01:52:08.141688 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:08.141654    2578 generic.go:356] "Generic (PLEG): container finished" podID="c4fedf30-2aba-447a-9db5-e3c36e73bfd4" containerID="10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c" exitCode=137
Sep 17 01:52:08.141832 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:08.141705    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-a14ec7"
Sep 17 01:52:08.141832 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:08.141703    2578 scope.go:117] "RemoveContainer" containerID="10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c"
Sep 17 01:52:08.150228 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:08.150214    2578 scope.go:117] "RemoveContainer" containerID="10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c"
Sep 17 01:52:08.150502 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:52:08.150479    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c\": container with ID starting with 10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c not found: ID does not exist" containerID="10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c"
Sep 17 01:52:08.150598 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:08.150515    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c"} err="failed to get container status \"10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c\": rpc error: code = NotFound desc = could not find container \"10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c\": container with ID starting with 10f5c366682e79dbb4e82f148486ed303aa1b834c4aa541fc96c7265e69aa55c not found: ID does not exist"
Sep 17 01:52:13.339823 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.339800    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:52:13.433203 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433117    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-credential-socket\") pod \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") "
Sep 17 01:52:13.433203 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433168    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-token\") pod \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") "
Sep 17 01:52:13.433203 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istiod-ca-cert\") pod \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") "
Sep 17 01:52:13.433468 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433228    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-socket\") pod \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") "
Sep 17 01:52:13.433468 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-certs\") pod \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") "
Sep 17 01:52:13.433468 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433277    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-envoy\") pod \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") "
Sep 17 01:52:13.433468 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433346    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-data\") pod \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") "
Sep 17 01:52:13.433468 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-podinfo\") pod \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") "
Sep 17 01:52:13.433468 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433414    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqxhd\" (UniqueName: \"kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-kube-api-access-kqxhd\") pod \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\" (UID: \"4ab276a1-7f0a-4be7-a0cb-06666d05dab6\") "
Sep 17 01:52:13.433468 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433448    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ab276a1-7f0a-4be7-a0cb-06666d05dab6" (UID: "4ab276a1-7f0a-4be7-a0cb-06666d05dab6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:52:13.433856 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433611    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ab276a1-7f0a-4be7-a0cb-06666d05dab6" (UID: "4ab276a1-7f0a-4be7-a0cb-06666d05dab6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:52:13.433856 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433620    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ab276a1-7f0a-4be7-a0cb-06666d05dab6" (UID: "4ab276a1-7f0a-4be7-a0cb-06666d05dab6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:52:13.433856 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433734    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ab276a1-7f0a-4be7-a0cb-06666d05dab6" (UID: "4ab276a1-7f0a-4be7-a0cb-06666d05dab6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:52:13.433856 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433762    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ab276a1-7f0a-4be7-a0cb-06666d05dab6" (UID: "4ab276a1-7f0a-4be7-a0cb-06666d05dab6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:52:13.433856 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433782    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-socket\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:13.433856 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433803    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-workload-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:13.433856 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.433817    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-credential-socket\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:13.435584 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.435554    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ab276a1-7f0a-4be7-a0cb-06666d05dab6" (UID: "4ab276a1-7f0a-4be7-a0cb-06666d05dab6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:52:13.435584 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.435570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-kube-api-access-kqxhd" (OuterVolumeSpecName: "kube-api-access-kqxhd") pod "4ab276a1-7f0a-4be7-a0cb-06666d05dab6" (UID: "4ab276a1-7f0a-4be7-a0cb-06666d05dab6"). InnerVolumeSpecName "kube-api-access-kqxhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:52:13.435993 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.435975    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ab276a1-7f0a-4be7-a0cb-06666d05dab6" (UID: "4ab276a1-7f0a-4be7-a0cb-06666d05dab6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:52:13.436044 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.435994    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ab276a1-7f0a-4be7-a0cb-06666d05dab6" (UID: "4ab276a1-7f0a-4be7-a0cb-06666d05dab6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 01:52:13.535299 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.535261    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-data\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:13.535299 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.535294    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-podinfo\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:13.535299 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.535303    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqxhd\" (UniqueName: \"kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-kube-api-access-kqxhd\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:13.535572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.535316    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-token\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:13.535572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.535325    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istiod-ca-cert\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:13.535572 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:13.535333    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab276a1-7f0a-4be7-a0cb-06666d05dab6-istio-envoy\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:14.175489 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.175455    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerID="2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f" exitCode=0
Sep 17 01:52:14.175678 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.175516    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"
Sep 17 01:52:14.175678 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.175519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" event={"ID":"4ab276a1-7f0a-4be7-a0cb-06666d05dab6","Type":"ContainerDied","Data":"2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f"}
Sep 17 01:52:14.175678 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.175644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4" event={"ID":"4ab276a1-7f0a-4be7-a0cb-06666d05dab6","Type":"ContainerDied","Data":"9581307066a8e82f5fcd8c4c083af33c948b67310c6a4376932a4a59ad86e89e"}
Sep 17 01:52:14.175678 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.175666    2578 scope.go:117] "RemoveContainer" containerID="2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f"
Sep 17 01:52:14.184497 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.184478    2578 scope.go:117] "RemoveContainer" containerID="7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643"
Sep 17 01:52:14.192620 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.192593    2578 scope.go:117] "RemoveContainer" containerID="2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f"
Sep 17 01:52:14.192961 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:52:14.192927    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f\": container with ID starting with 2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f not found: ID does not exist" containerID="2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f"
Sep 17 01:52:14.193063 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.192965    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f"} err="failed to get container status \"2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f\": rpc error: code = NotFound desc = could not find container \"2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f\": container with ID starting with 2059a17a7722dc63ef6fcdf6fe54e0730f9ad565af4f713b1927b017258d476f not found: ID does not exist"
Sep 17 01:52:14.193063 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.192991    2578 scope.go:117] "RemoveContainer" containerID="7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643"
Sep 17 01:52:14.193307 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:52:14.193289    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643\": container with ID starting with 7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643 not found: ID does not exist" containerID="7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643"
Sep 17 01:52:14.193356 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.193314    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643"} err="failed to get container status \"7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643\": rpc error: code = NotFound desc = could not find container \"7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643\": container with ID starting with 7441045f5a4cd80ca0f5aac62eed5c71b87b9dab56ff9dc110e133c39991d643 not found: ID does not exist"
Sep 17 01:52:14.194414 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.194393    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"]
Sep 17 01:52:14.198117 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:14.198095    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6f3c6f-openshift-default-56584f67c8-bdqp4"]
Sep 17 01:52:15.920982 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:15.920949    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" path="/var/lib/kubelet/pods/4ab276a1-7f0a-4be7-a0cb-06666d05dab6/volumes"
Sep 17 01:52:18.865740 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.865703    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"]
Sep 17 01:52:18.866268 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866247    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29bc7957-1077-4bec-a369-c99af84e5468" containerName="payload-pre-processing"
Sep 17 01:52:18.866268 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866267    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="29bc7957-1077-4bec-a369-c99af84e5468" containerName="payload-pre-processing"
Sep 17 01:52:18.866405 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866282    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4fedf30-2aba-447a-9db5-e3c36e73bfd4" containerName="test-curl-10047-a14ec7"
Sep 17 01:52:18.866405 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866291    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4fedf30-2aba-447a-9db5-e3c36e73bfd4" containerName="test-curl-10047-a14ec7"
Sep 17 01:52:18.866405 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866311    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy"
Sep 17 01:52:18.866405 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866321    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy"
Sep 17 01:52:18.866405 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866354    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56de9ca-1bc5-4053-b2c1-d8a79935f588" containerName="payload-processing"
Sep 17 01:52:18.866405 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866363    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56de9ca-1bc5-4053-b2c1-d8a79935f588" containerName="payload-processing"
Sep 17 01:52:18.866725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866456    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="29bc7957-1077-4bec-a369-c99af84e5468" containerName="payload-pre-processing"
Sep 17 01:52:18.866725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866469    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy"
Sep 17 01:52:18.866725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866482    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4fedf30-2aba-447a-9db5-e3c36e73bfd4" containerName="test-curl-10047-a14ec7"
Sep 17 01:52:18.866725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.866495    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f56de9ca-1bc5-4053-b2c1-d8a79935f588" containerName="payload-processing"
Sep 17 01:52:18.872317 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.872295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.876409 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.876382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-fcd402\"/\"rate-test-model-fcd402-kserve-self-signed-certs\""
Sep 17 01:52:18.879032 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.879008    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"]
Sep 17 01:52:18.885174 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.885150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-home\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.885291 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.885187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-tmp-dir\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.885291 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.885215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-dshm\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.885370 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.885320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8918bcc-0ff7-4eda-b411-422d5670de16-tls-certs\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.885409 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.885366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-model-cache\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.885462 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.885439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zg5g\" (UniqueName: \"kubernetes.io/projected/b8918bcc-0ff7-4eda-b411-422d5670de16-kube-api-access-9zg5g\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.986739 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.986710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-model-cache\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.986931 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.986750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zg5g\" (UniqueName: \"kubernetes.io/projected/b8918bcc-0ff7-4eda-b411-422d5670de16-kube-api-access-9zg5g\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.986931 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.986799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-home\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.986931 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.986827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-tmp-dir\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.986931 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.986852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-dshm\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.986931 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.986879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8918bcc-0ff7-4eda-b411-422d5670de16-tls-certs\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.987196 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.987171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-model-cache\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.987243 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.987209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-home\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.987282 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.987240    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-tmp-dir\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.989119 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.989098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-dshm\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.989311 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.989295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8918bcc-0ff7-4eda-b411-422d5670de16-tls-certs\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:18.994613 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:18.994590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zg5g\" (UniqueName: \"kubernetes.io/projected/b8918bcc-0ff7-4eda-b411-422d5670de16-kube-api-access-9zg5g\") pod \"rate-test-model-fcd402-kserve-74888974c9-n2wzm\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") " pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:19.183287 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:19.183189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:19.488401 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:19.488374    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"]
Sep 17 01:52:19.490471 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:52:19.490439    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8918bcc_0ff7_4eda_b411_422d5670de16.slice/crio-c9af637ee57d3fa3a7f52cd9b9cb5d269325f67663c23ff8fd97dbbe81c1e65e WatchSource:0}: Error finding container c9af637ee57d3fa3a7f52cd9b9cb5d269325f67663c23ff8fd97dbbe81c1e65e: Status 404 returned error can't find the container with id c9af637ee57d3fa3a7f52cd9b9cb5d269325f67663c23ff8fd97dbbe81c1e65e
Sep 17 01:52:20.208089 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.208047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm" event={"ID":"b8918bcc-0ff7-4eda-b411-422d5670de16","Type":"ContainerStarted","Data":"d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b"}
Sep 17 01:52:20.208509 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.208098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm" event={"ID":"b8918bcc-0ff7-4eda-b411-422d5670de16","Type":"ContainerStarted","Data":"c9af637ee57d3fa3a7f52cd9b9cb5d269325f67663c23ff8fd97dbbe81c1e65e"}
Sep 17 01:52:20.225678 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.225617    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm" podStartSLOduration=2.2255984460000002 podStartE2EDuration="2.225598446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:52:20.224189514 +0000 UTC m=+1818.894705603" watchObservedRunningTime="2026-09-17 01:52:20.225598446 +0000 UTC m=+1818.896114536"
Sep 17 01:52:20.637194 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.637157    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd" podUID="6ad0d16e-c06e-4341-8cb4-51908394bf61" containerName="main" containerID="cri-o://a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df" gracePeriod=45
Sep 17 01:52:20.880945 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.880919    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:52:20.904358 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.904288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-dshm\") pod \"6ad0d16e-c06e-4341-8cb4-51908394bf61\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") "
Sep 17 01:52:20.904358 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.904347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad0d16e-c06e-4341-8cb4-51908394bf61-tls-certs\") pod \"6ad0d16e-c06e-4341-8cb4-51908394bf61\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") "
Sep 17 01:52:20.904595 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.904372    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzx6z\" (UniqueName: \"kubernetes.io/projected/6ad0d16e-c06e-4341-8cb4-51908394bf61-kube-api-access-zzx6z\") pod \"6ad0d16e-c06e-4341-8cb4-51908394bf61\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") "
Sep 17 01:52:20.904595 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.904391    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-home\") pod \"6ad0d16e-c06e-4341-8cb4-51908394bf61\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") "
Sep 17 01:52:20.904595 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.904480    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-model-cache\") pod \"6ad0d16e-c06e-4341-8cb4-51908394bf61\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") "
Sep 17 01:52:20.904595 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.904519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-tmp-dir\") pod \"6ad0d16e-c06e-4341-8cb4-51908394bf61\" (UID: \"6ad0d16e-c06e-4341-8cb4-51908394bf61\") "
Sep 17 01:52:20.904831 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.904788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-home" (OuterVolumeSpecName: "home") pod "6ad0d16e-c06e-4341-8cb4-51908394bf61" (UID: "6ad0d16e-c06e-4341-8cb4-51908394bf61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:52:20.904831 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.904813    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ad0d16e-c06e-4341-8cb4-51908394bf61" (UID: "6ad0d16e-c06e-4341-8cb4-51908394bf61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:52:20.904972 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.904954    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ad0d16e-c06e-4341-8cb4-51908394bf61" (UID: "6ad0d16e-c06e-4341-8cb4-51908394bf61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:52:20.906701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.906675    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad0d16e-c06e-4341-8cb4-51908394bf61-kube-api-access-zzx6z" (OuterVolumeSpecName: "kube-api-access-zzx6z") pod "6ad0d16e-c06e-4341-8cb4-51908394bf61" (UID: "6ad0d16e-c06e-4341-8cb4-51908394bf61"). InnerVolumeSpecName "kube-api-access-zzx6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:52:20.906930 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.906915    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-dshm" (OuterVolumeSpecName: "dshm") pod "6ad0d16e-c06e-4341-8cb4-51908394bf61" (UID: "6ad0d16e-c06e-4341-8cb4-51908394bf61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:52:20.906994 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:20.906933    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad0d16e-c06e-4341-8cb4-51908394bf61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ad0d16e-c06e-4341-8cb4-51908394bf61" (UID: "6ad0d16e-c06e-4341-8cb4-51908394bf61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:52:21.005438 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.005405    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:21.005438 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.005431    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:21.005438 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.005440    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad0d16e-c06e-4341-8cb4-51908394bf61-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:21.005682 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.005449    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzx6z\" (UniqueName: \"kubernetes.io/projected/6ad0d16e-c06e-4341-8cb4-51908394bf61-kube-api-access-zzx6z\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:21.005682 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.005459    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:21.005682 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.005467    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad0d16e-c06e-4341-8cb4-51908394bf61-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:52:21.214330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.214245    2578 generic.go:356] "Generic (PLEG): container finished" podID="6ad0d16e-c06e-4341-8cb4-51908394bf61" containerID="a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df" exitCode=0
Sep 17 01:52:21.214330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.214280    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"
Sep 17 01:52:21.214330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.214314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd" event={"ID":"6ad0d16e-c06e-4341-8cb4-51908394bf61","Type":"ContainerDied","Data":"a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df"}
Sep 17 01:52:21.214823 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.214347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd" event={"ID":"6ad0d16e-c06e-4341-8cb4-51908394bf61","Type":"ContainerDied","Data":"50335a51d1f6c6af09a84c30d8b0b0b704c1fc8c129f50c8e6e6c08145601b28"}
Sep 17 01:52:21.214823 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.214366    2578 scope.go:117] "RemoveContainer" containerID="a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df"
Sep 17 01:52:21.223100 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.223082    2578 scope.go:117] "RemoveContainer" containerID="a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df"
Sep 17 01:52:21.223363 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:52:21.223346    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df\": container with ID starting with a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df not found: ID does not exist" containerID="a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df"
Sep 17 01:52:21.223402 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.223370    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df"} err="failed to get container status \"a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df\": rpc error: code = NotFound desc = could not find container \"a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df\": container with ID starting with a99d3e736cd76a5321ae50812e6f0f698abeb1eedf11925cd5efa06e2f30f8df not found: ID does not exist"
Sep 17 01:52:21.235680 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.235654    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"]
Sep 17 01:52:21.240263 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.240235    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/sub-test-model-fcd402-kserve-5df5bf4745-6pzgd"]
Sep 17 01:52:21.920240 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:21.920199    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad0d16e-c06e-4341-8cb4-51908394bf61" path="/var/lib/kubelet/pods/6ad0d16e-c06e-4341-8cb4-51908394bf61/volumes"
Sep 17 01:52:29.183466 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:29.183429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:29.183888 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:29.183487    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:29.196648 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:29.196624    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:29.267743 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:29.267709    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:52:44.068285 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.068249    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"]
Sep 17 01:52:44.068730 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.068641    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy"
Sep 17 01:52:44.068730 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.068653    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy"
Sep 17 01:52:44.068730 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.068662    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad0d16e-c06e-4341-8cb4-51908394bf61" containerName="main"
Sep 17 01:52:44.068730 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.068668    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad0d16e-c06e-4341-8cb4-51908394bf61" containerName="main"
Sep 17 01:52:44.068730 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.068727    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab276a1-7f0a-4be7-a0cb-06666d05dab6" containerName="istio-proxy"
Sep 17 01:52:44.068730 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.068736    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad0d16e-c06e-4341-8cb4-51908394bf61" containerName="main"
Sep 17 01:52:44.071778 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.071761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.075203 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.075179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-361849\"/\"default-dockercfg-vqzvr\""
Sep 17 01:52:44.075367 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.075228    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-361849\"/\"rate-test-model-361849-kserve-self-signed-certs\""
Sep 17 01:52:44.075367 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.075262    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-361849\"/\"kube-root-ca.crt\""
Sep 17 01:52:44.075367 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.075281    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-361849\"/\"openshift-service-ca.crt\""
Sep 17 01:52:44.081209 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.081184    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"]
Sep 17 01:52:44.234338 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.234303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-home\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.234483 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.234391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-tmp-dir\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.234483 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.234465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6926abe-957d-4908-8761-1fd779e7693c-tls-certs\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.234589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.234492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-model-cache\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.234589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.234524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmtp\" (UniqueName: \"kubernetes.io/projected/d6926abe-957d-4908-8761-1fd779e7693c-kube-api-access-7fmtp\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.234589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.234585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-dshm\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.335873 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.335785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-home\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.335873 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.335845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-tmp-dir\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.336109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.335913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6926abe-957d-4908-8761-1fd779e7693c-tls-certs\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.336109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.335932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-model-cache\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.336109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.335964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmtp\" (UniqueName: \"kubernetes.io/projected/d6926abe-957d-4908-8761-1fd779e7693c-kube-api-access-7fmtp\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.336109 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.335996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-dshm\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.336386 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.336243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-tmp-dir\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.336386 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.336311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-home\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.336386 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.336372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-model-cache\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.338220 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.338202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-dshm\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.338491 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.338474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6926abe-957d-4908-8761-1fd779e7693c-tls-certs\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.343699 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.343672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmtp\" (UniqueName: \"kubernetes.io/projected/d6926abe-957d-4908-8761-1fd779e7693c-kube-api-access-7fmtp\") pod \"rate-test-model-361849-kserve-78694dc96c-lg5tv\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") " pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.381490 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.381463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:44.652687 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:44.652661    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"]
Sep 17 01:52:44.655093 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:52:44.655068    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6926abe_957d_4908_8761_1fd779e7693c.slice/crio-41163d3b566a8d7047abea5e808ab0598fd3eb7dc5b9148974596b21dbdbb123 WatchSource:0}: Error finding container 41163d3b566a8d7047abea5e808ab0598fd3eb7dc5b9148974596b21dbdbb123: Status 404 returned error can't find the container with id 41163d3b566a8d7047abea5e808ab0598fd3eb7dc5b9148974596b21dbdbb123
Sep 17 01:52:45.336823 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:45.336783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv" event={"ID":"d6926abe-957d-4908-8761-1fd779e7693c","Type":"ContainerStarted","Data":"f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb"}
Sep 17 01:52:45.337233 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:45.336836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv" event={"ID":"d6926abe-957d-4908-8761-1fd779e7693c","Type":"ContainerStarted","Data":"41163d3b566a8d7047abea5e808ab0598fd3eb7dc5b9148974596b21dbdbb123"}
Sep 17 01:52:45.353595 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:45.353523    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv" podStartSLOduration=1.353506525 podStartE2EDuration="1.353506525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:52:45.351944522 +0000 UTC m=+1844.022460623" watchObservedRunningTime="2026-09-17 01:52:45.353506525 +0000 UTC m=+1844.024022614"
Sep 17 01:52:54.382063 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:54.382027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:54.382063 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:54.382071    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:54.394321 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:54.394295    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:52:55.396654 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:52:55.396626    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:53:24.671277 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:24.671197    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"]
Sep 17 01:53:25.671894 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:25.671861    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"]
Sep 17 01:53:26.364986 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.364951    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"]
Sep 17 01:53:26.368169 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.368135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.370591 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.370571    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 01:53:26.370708 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.370645    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-647268-openshift-default-dockercfg-ffwzk\""
Sep 17 01:53:26.379202 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.379176    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"]
Sep 17 01:53:26.436123 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.436074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-credential-socket\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.436293 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.436128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-socket\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.436293 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.436178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-envoy\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.436293 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.436204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-data\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.436293 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.436270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-podinfo\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.436470 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.436301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-certs\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.436470 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.436328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-token\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.436470 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.436381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istiod-ca-cert\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.436606 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.436482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftkt\" (UniqueName: \"kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-kube-api-access-pftkt\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.536919 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.536882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-podinfo\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537120 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.536928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-certs\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537120 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.536979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-token\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537120 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istiod-ca-cert\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537120 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftkt\" (UniqueName: \"kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-kube-api-access-pftkt\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537346 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-credential-socket\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537346 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-socket\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537346 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-envoy\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537346 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-data\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537579 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-certs\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537646 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-socket\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537734 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537712    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-credential-socket\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.537970 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-data\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.538045 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.537979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istiod-ca-cert\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.539446 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.539427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-envoy\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.539593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.539574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-podinfo\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.544666 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.544637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-token\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.545459 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.545437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftkt\" (UniqueName: \"kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-kube-api-access-pftkt\") pod \"e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") " pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.681859 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.681760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:26.965361 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.965325    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"]
Sep 17 01:53:26.968747 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.968633    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:53:26.968747 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.968727    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:53:26.969619 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:26.968940    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:53:27.536059 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:27.536025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" event={"ID":"2a457cf9-d9c1-489d-b589-9afb8edeb52c","Type":"ContainerStarted","Data":"fbd3646ad744d4c7172f3dae241314accef459fcf9ba0b3ea38195a8e7eb1a94"}
Sep 17 01:53:27.536245 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:27.536079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" event={"ID":"2a457cf9-d9c1-489d-b589-9afb8edeb52c","Type":"ContainerStarted","Data":"5f48fb74e88f22a13124103b67af17eaa2f36e3b3b6d6677991aa5019cf45395"}
Sep 17 01:53:27.552936 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:27.552879    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" podStartSLOduration=1.552860002 podStartE2EDuration="1.552860002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:53:27.552763161 +0000 UTC m=+1886.223279275" watchObservedRunningTime="2026-09-17 01:53:27.552860002 +0000 UTC m=+1886.223376092"
Sep 17 01:53:27.682701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:27.682668    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:28.687199 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:28.687167    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:29.546174 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:29.546140    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:29.547195 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:29.547173    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:31.747350 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.747317    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"]
Sep 17 01:53:31.751094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.751076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:53:31.753330 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.753305    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-647268-dockercfg-q6rps\""
Sep 17 01:53:31.753437 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.753314    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-647268\""
Sep 17 01:53:31.762804 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.762775    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"]
Sep 17 01:53:31.780021 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.779991    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"]
Sep 17 01:53:31.784270 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.784232    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:53:31.784846 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.784808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38380747-222c-4dbe-a1d4-5bee2d1761f8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms\" (UID: \"38380747-222c-4dbe-a1d4-5bee2d1761f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:53:31.784957 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.784934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54nfq\" (UniqueName: \"kubernetes.io/projected/38380747-222c-4dbe-a1d4-5bee2d1761f8-kube-api-access-54nfq\") pod \"payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms\" (UID: \"38380747-222c-4dbe-a1d4-5bee2d1761f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:53:31.790106 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.790078    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"]
Sep 17 01:53:31.885523 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.885487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38380747-222c-4dbe-a1d4-5bee2d1761f8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms\" (UID: \"38380747-222c-4dbe-a1d4-5bee2d1761f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:53:31.885753 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.885574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fj4t\" (UniqueName: \"kubernetes.io/projected/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-kube-api-access-6fj4t\") pod \"payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69\" (UID: \"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:53:31.885753 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.885611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54nfq\" (UniqueName: \"kubernetes.io/projected/38380747-222c-4dbe-a1d4-5bee2d1761f8-kube-api-access-54nfq\") pod \"payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms\" (UID: \"38380747-222c-4dbe-a1d4-5bee2d1761f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:53:31.885753 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.885675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69\" (UID: \"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:53:31.886457 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.886436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38380747-222c-4dbe-a1d4-5bee2d1761f8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms\" (UID: \"38380747-222c-4dbe-a1d4-5bee2d1761f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:53:31.895267 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.895236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54nfq\" (UniqueName: \"kubernetes.io/projected/38380747-222c-4dbe-a1d4-5bee2d1761f8-kube-api-access-54nfq\") pod \"payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms\" (UID: \"38380747-222c-4dbe-a1d4-5bee2d1761f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:53:31.986584 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.986521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fj4t\" (UniqueName: \"kubernetes.io/projected/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-kube-api-access-6fj4t\") pod \"payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69\" (UID: \"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:53:31.986840 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.986812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69\" (UID: \"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:53:31.987442 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.987423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69\" (UID: \"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:53:31.994204 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:31.994177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fj4t\" (UniqueName: \"kubernetes.io/projected/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-kube-api-access-6fj4t\") pod \"payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69\" (UID: \"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:53:32.062934 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.062907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:53:32.095575 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.095544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:53:32.336478 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.336450    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"]
Sep 17 01:53:32.338605 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:53:32.338573    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38380747_222c_4dbe_a1d4_5bee2d1761f8.slice/crio-d0dfbdcdc348e46a1f146b330a21bf66d4d1a640bf11a4fc97c4d6c775edd453 WatchSource:0}: Error finding container d0dfbdcdc348e46a1f146b330a21bf66d4d1a640bf11a4fc97c4d6c775edd453: Status 404 returned error can't find the container with id d0dfbdcdc348e46a1f146b330a21bf66d4d1a640bf11a4fc97c4d6c775edd453
Sep 17 01:53:32.373932 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.373905    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"]
Sep 17 01:53:32.377904 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:53:32.377864    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe0a9a5_0f94_4f6b_8f66_bb84bda3ef38.slice/crio-ea19b084c3c7b0b77352ee40cad4830fe76717d5a9981f3d92072690c8095bc5 WatchSource:0}: Error finding container ea19b084c3c7b0b77352ee40cad4830fe76717d5a9981f3d92072690c8095bc5: Status 404 returned error can't find the container with id ea19b084c3c7b0b77352ee40cad4830fe76717d5a9981f3d92072690c8095bc5
Sep 17 01:53:32.563023 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.562988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69" event={"ID":"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38","Type":"ContainerStarted","Data":"d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c"}
Sep 17 01:53:32.563224 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.563047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69" event={"ID":"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38","Type":"ContainerStarted","Data":"ea19b084c3c7b0b77352ee40cad4830fe76717d5a9981f3d92072690c8095bc5"}
Sep 17 01:53:32.563224 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.563076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:53:32.564548 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.564494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms" event={"ID":"38380747-222c-4dbe-a1d4-5bee2d1761f8","Type":"ContainerStarted","Data":"ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0"}
Sep 17 01:53:32.564669 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.564585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms" event={"ID":"38380747-222c-4dbe-a1d4-5bee2d1761f8","Type":"ContainerStarted","Data":"d0dfbdcdc348e46a1f146b330a21bf66d4d1a640bf11a4fc97c4d6c775edd453"}
Sep 17 01:53:32.564669 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.564615    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:53:32.579127 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:32.579071    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69" podStartSLOduration=1.5790524879999999 podStartE2EDuration="1.579052488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:53:32.577848886 +0000 UTC m=+1891.248364977" watchObservedRunningTime="2026-09-17 01:53:32.579052488 +0000 UTC m=+1891.249568578"
Sep 17 01:53:39.692360 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:39.692274    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm" podUID="b8918bcc-0ff7-4eda-b411-422d5670de16" containerName="main" containerID="cri-o://d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b" gracePeriod=45
Sep 17 01:53:39.985757 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:39.985392    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:53:40.005117 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.005066    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms" podStartSLOduration=9.005050605 podStartE2EDuration="9.005050605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:53:32.594332605 +0000 UTC m=+1891.264848698" watchObservedRunningTime="2026-09-17 01:53:40.005050605 +0000 UTC m=+1898.675566729"
Sep 17 01:53:40.085878 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.085839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zg5g\" (UniqueName: \"kubernetes.io/projected/b8918bcc-0ff7-4eda-b411-422d5670de16-kube-api-access-9zg5g\") pod \"b8918bcc-0ff7-4eda-b411-422d5670de16\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") "
Sep 17 01:53:40.086144 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.086127    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8918bcc-0ff7-4eda-b411-422d5670de16-tls-certs\") pod \"b8918bcc-0ff7-4eda-b411-422d5670de16\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") "
Sep 17 01:53:40.086266 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.086253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-tmp-dir\") pod \"b8918bcc-0ff7-4eda-b411-422d5670de16\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") "
Sep 17 01:53:40.086496 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.086480    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-model-cache\") pod \"b8918bcc-0ff7-4eda-b411-422d5670de16\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") "
Sep 17 01:53:40.086725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.086707    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-home\") pod \"b8918bcc-0ff7-4eda-b411-422d5670de16\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") "
Sep 17 01:53:40.086860 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.086847    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-dshm\") pod \"b8918bcc-0ff7-4eda-b411-422d5670de16\" (UID: \"b8918bcc-0ff7-4eda-b411-422d5670de16\") "
Sep 17 01:53:40.087121 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.087091    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8918bcc-0ff7-4eda-b411-422d5670de16" (UID: "b8918bcc-0ff7-4eda-b411-422d5670de16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:53:40.087263 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.087241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8918bcc-0ff7-4eda-b411-422d5670de16" (UID: "b8918bcc-0ff7-4eda-b411-422d5670de16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:53:40.087466 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.087448    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:40.087590 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.087574    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:40.089660 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.087769    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-home" (OuterVolumeSpecName: "home") pod "b8918bcc-0ff7-4eda-b411-422d5670de16" (UID: "b8918bcc-0ff7-4eda-b411-422d5670de16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:53:40.090675 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.090654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8918bcc-0ff7-4eda-b411-422d5670de16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8918bcc-0ff7-4eda-b411-422d5670de16" (UID: "b8918bcc-0ff7-4eda-b411-422d5670de16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:53:40.093028 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.092994    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-dshm" (OuterVolumeSpecName: "dshm") pod "b8918bcc-0ff7-4eda-b411-422d5670de16" (UID: "b8918bcc-0ff7-4eda-b411-422d5670de16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:53:40.093142 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.093029    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8918bcc-0ff7-4eda-b411-422d5670de16-kube-api-access-9zg5g" (OuterVolumeSpecName: "kube-api-access-9zg5g") pod "b8918bcc-0ff7-4eda-b411-422d5670de16" (UID: "b8918bcc-0ff7-4eda-b411-422d5670de16"). InnerVolumeSpecName "kube-api-access-9zg5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:53:40.189167 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.189118    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8918bcc-0ff7-4eda-b411-422d5670de16-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:40.189167 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.189165    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:40.189419 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.189179    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8918bcc-0ff7-4eda-b411-422d5670de16-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:40.189419 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.189193    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zg5g\" (UniqueName: \"kubernetes.io/projected/b8918bcc-0ff7-4eda-b411-422d5670de16-kube-api-access-9zg5g\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:40.607382 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.607345    2578 generic.go:356] "Generic (PLEG): container finished" podID="b8918bcc-0ff7-4eda-b411-422d5670de16" containerID="d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b" exitCode=0
Sep 17 01:53:40.607593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.607391    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"
Sep 17 01:53:40.607593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.607432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm" event={"ID":"b8918bcc-0ff7-4eda-b411-422d5670de16","Type":"ContainerDied","Data":"d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b"}
Sep 17 01:53:40.607593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.607480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm" event={"ID":"b8918bcc-0ff7-4eda-b411-422d5670de16","Type":"ContainerDied","Data":"c9af637ee57d3fa3a7f52cd9b9cb5d269325f67663c23ff8fd97dbbe81c1e65e"}
Sep 17 01:53:40.607593 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.607502    2578 scope.go:117] "RemoveContainer" containerID="d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b"
Sep 17 01:53:40.617041 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.617026    2578 scope.go:117] "RemoveContainer" containerID="d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b"
Sep 17 01:53:40.617316 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:53:40.617296    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b\": container with ID starting with d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b not found: ID does not exist" containerID="d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b"
Sep 17 01:53:40.617370 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.617325    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b"} err="failed to get container status \"d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b\": rpc error: code = NotFound desc = could not find container \"d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b\": container with ID starting with d58a52f627d90bca01511edbde5f0b8df428e6602e8e19def42056d0d7cd550b not found: ID does not exist"
Sep 17 01:53:40.628186 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.628153    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"]
Sep 17 01:53:40.631492 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.631470    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd402/rate-test-model-fcd402-kserve-74888974c9-n2wzm"]
Sep 17 01:53:40.693136 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.693070    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv" podUID="d6926abe-957d-4908-8761-1fd779e7693c" containerName="main" containerID="cri-o://f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb" gracePeriod=45
Sep 17 01:53:40.949318 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:40.949295    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:53:41.098875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.098835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-tmp-dir\") pod \"d6926abe-957d-4908-8761-1fd779e7693c\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") "
Sep 17 01:53:41.098875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.098885    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-home\") pod \"d6926abe-957d-4908-8761-1fd779e7693c\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") "
Sep 17 01:53:41.099128 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.098951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-dshm\") pod \"d6926abe-957d-4908-8761-1fd779e7693c\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") "
Sep 17 01:53:41.099128 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.099002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fmtp\" (UniqueName: \"kubernetes.io/projected/d6926abe-957d-4908-8761-1fd779e7693c-kube-api-access-7fmtp\") pod \"d6926abe-957d-4908-8761-1fd779e7693c\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") "
Sep 17 01:53:41.099128 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.099072    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6926abe-957d-4908-8761-1fd779e7693c-tls-certs\") pod \"d6926abe-957d-4908-8761-1fd779e7693c\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") "
Sep 17 01:53:41.099128 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.099090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6926abe-957d-4908-8761-1fd779e7693c" (UID: "d6926abe-957d-4908-8761-1fd779e7693c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:53:41.099347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.099132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-model-cache\") pod \"d6926abe-957d-4908-8761-1fd779e7693c\" (UID: \"d6926abe-957d-4908-8761-1fd779e7693c\") "
Sep 17 01:53:41.099347 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.099184    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-home" (OuterVolumeSpecName: "home") pod "d6926abe-957d-4908-8761-1fd779e7693c" (UID: "d6926abe-957d-4908-8761-1fd779e7693c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:53:41.099453 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.099388    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6926abe-957d-4908-8761-1fd779e7693c" (UID: "d6926abe-957d-4908-8761-1fd779e7693c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:53:41.099453 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.099439    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:41.099453 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.099452    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:41.101157 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.101131    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6926abe-957d-4908-8761-1fd779e7693c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6926abe-957d-4908-8761-1fd779e7693c" (UID: "d6926abe-957d-4908-8761-1fd779e7693c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:53:41.101282 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.101182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-dshm" (OuterVolumeSpecName: "dshm") pod "d6926abe-957d-4908-8761-1fd779e7693c" (UID: "d6926abe-957d-4908-8761-1fd779e7693c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:53:41.101282 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.101202    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6926abe-957d-4908-8761-1fd779e7693c-kube-api-access-7fmtp" (OuterVolumeSpecName: "kube-api-access-7fmtp") pod "d6926abe-957d-4908-8761-1fd779e7693c" (UID: "d6926abe-957d-4908-8761-1fd779e7693c"). InnerVolumeSpecName "kube-api-access-7fmtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:53:41.200344 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.200244    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:41.200344 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.200277    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fmtp\" (UniqueName: \"kubernetes.io/projected/d6926abe-957d-4908-8761-1fd779e7693c-kube-api-access-7fmtp\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:41.200344 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.200290    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6926abe-957d-4908-8761-1fd779e7693c-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:41.200344 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.200304    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6926abe-957d-4908-8761-1fd779e7693c-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:53:41.613668 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.613633    2578 generic.go:356] "Generic (PLEG): container finished" podID="d6926abe-957d-4908-8761-1fd779e7693c" containerID="f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb" exitCode=0
Sep 17 01:53:41.613861 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.613677    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"
Sep 17 01:53:41.613861 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.613718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv" event={"ID":"d6926abe-957d-4908-8761-1fd779e7693c","Type":"ContainerDied","Data":"f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb"}
Sep 17 01:53:41.613861 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.613766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv" event={"ID":"d6926abe-957d-4908-8761-1fd779e7693c","Type":"ContainerDied","Data":"41163d3b566a8d7047abea5e808ab0598fd3eb7dc5b9148974596b21dbdbb123"}
Sep 17 01:53:41.613861 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.613784    2578 scope.go:117] "RemoveContainer" containerID="f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb"
Sep 17 01:53:41.622530 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.622513    2578 scope.go:117] "RemoveContainer" containerID="f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb"
Sep 17 01:53:41.622803 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:53:41.622786    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb\": container with ID starting with f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb not found: ID does not exist" containerID="f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb"
Sep 17 01:53:41.622856 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.622812    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb"} err="failed to get container status \"f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb\": rpc error: code = NotFound desc = could not find container \"f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb\": container with ID starting with f5f5a3880faa0fdefd385d96664457b8dcef1f746bc1849cac5d2452879f13cb not found: ID does not exist"
Sep 17 01:53:41.633808 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.633785    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"]
Sep 17 01:53:41.635508 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.635484    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-361849/rate-test-model-361849-kserve-78694dc96c-lg5tv"]
Sep 17 01:53:41.920441 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.920347    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8918bcc-0ff7-4eda-b411-422d5670de16" path="/var/lib/kubelet/pods/b8918bcc-0ff7-4eda-b411-422d5670de16/volumes"
Sep 17 01:53:41.920968 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:41.920947    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6926abe-957d-4908-8761-1fd779e7693c" path="/var/lib/kubelet/pods/d6926abe-957d-4908-8761-1fd779e7693c/volumes"
Sep 17 01:53:43.571281 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:43.571257    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:53:43.571666 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:43.571318    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:53:45.587648 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:45.587617    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 01:53:45.588092 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:45.587665    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 01:53:46.642133 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:46.642098    2578 generic.go:356] "Generic (PLEG): container finished" podID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerID="fbd3646ad744d4c7172f3dae241314accef459fcf9ba0b3ea38195a8e7eb1a94" exitCode=0
Sep 17 01:53:46.642600 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:46.642169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" event={"ID":"2a457cf9-d9c1-489d-b589-9afb8edeb52c","Type":"ContainerDied","Data":"fbd3646ad744d4c7172f3dae241314accef459fcf9ba0b3ea38195a8e7eb1a94"}
Sep 17 01:53:46.642600 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:46.642573    2578 scope.go:117] "RemoveContainer" containerID="fbd3646ad744d4c7172f3dae241314accef459fcf9ba0b3ea38195a8e7eb1a94"
Sep 17 01:53:46.643687 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:46.643658    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:53:46.643745 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:46.643729    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:53:46.643781 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:46.643766    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:53:47.650169 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:47.650139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" event={"ID":"2a457cf9-d9c1-489d-b589-9afb8edeb52c","Type":"ContainerStarted","Data":"38af956206293518f1ec86583ac9bb66d2c71704e5237b95280f7cea99483a23"}
Sep 17 01:53:47.683142 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:47.682740    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:47.683142 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:47.682859    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 17 01:53:47.683142 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:47.682900    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 01:53:48.682907 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:48.682866    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 17 01:53:48.683403 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:48.682951    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 01:53:49.683344 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:49.682909    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 17 01:53:49.683344 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:49.682974    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 01:53:50.683117 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:50.683071    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 17 01:53:50.683329 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:50.683141    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 01:53:51.682967 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:51.682926    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 17 01:53:51.683458 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:51.683016    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 01:53:52.682302 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:52.682267    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 17 01:53:52.682489 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:52.682327    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 01:53:53.686229 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:53.686201    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:53.686616 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:53.686520    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:53.687125 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:53.687105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:53:55.240755 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.240724    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"]
Sep 17 01:53:55.241189 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.241171    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8918bcc-0ff7-4eda-b411-422d5670de16" containerName="main"
Sep 17 01:53:55.241189 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.241189    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8918bcc-0ff7-4eda-b411-422d5670de16" containerName="main"
Sep 17 01:53:55.241327 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.241210    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6926abe-957d-4908-8761-1fd779e7693c" containerName="main"
Sep 17 01:53:55.241327 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.241216    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6926abe-957d-4908-8761-1fd779e7693c" containerName="main"
Sep 17 01:53:55.241327 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.241317    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8918bcc-0ff7-4eda-b411-422d5670de16" containerName="main"
Sep 17 01:53:55.241511 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.241332    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6926abe-957d-4908-8761-1fd779e7693c" containerName="main"
Sep 17 01:53:55.244875 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.244856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.247570 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.247520    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-29e19c-openshift-default-dockercfg-6mthr\""
Sep 17 01:53:55.255583 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.255564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-credential-socket\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.255681 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.255595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-certs\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.255681 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.255633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-envoy\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.255681 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.255656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zk2h\" (UniqueName: \"kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-kube-api-access-8zk2h\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.255810 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.255687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7901ff35-db82-4476-8a59-acd18a605682-istiod-ca-cert\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.255810 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.255774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7901ff35-db82-4476-8a59-acd18a605682-istio-podinfo\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.255890 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.255826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-istio-token\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.255890 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.255870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-socket\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.255970 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.255932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-data\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.261750 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.261727    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"]
Sep 17 01:53:55.356989 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.356920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-envoy\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.356989 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.356963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zk2h\" (UniqueName: \"kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-kube-api-access-8zk2h\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.357394 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.357005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7901ff35-db82-4476-8a59-acd18a605682-istiod-ca-cert\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.357394 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.357059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7901ff35-db82-4476-8a59-acd18a605682-istio-podinfo\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.357394 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.357098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-istio-token\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.357394 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.357131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-socket\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.357394 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.357175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-data\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.357394 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.357243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-credential-socket\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.357394 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.357277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-certs\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.358078 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.358048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-socket\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.358179 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.358090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-certs\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.358232 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.358180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-data\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.358526 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.358498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7901ff35-db82-4476-8a59-acd18a605682-istiod-ca-cert\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.358622 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.358528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-credential-socket\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.359455 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.359435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-envoy\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.359961 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.359939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7901ff35-db82-4476-8a59-acd18a605682-istio-podinfo\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.364844 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.364820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-istio-token\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.364964 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.364948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zk2h\" (UniqueName: \"kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-kube-api-access-8zk2h\") pod \"e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") " pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.556021 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.555995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:55.829295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.829266    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"]
Sep 17 01:53:55.830940 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:53:55.830915    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7901ff35_db82_4476_8a59_acd18a605682.slice/crio-0804ba27d6f3cecd546d4b3b929c15399785fe5c5ab31d38280f6496e435fd16 WatchSource:0}: Error finding container 0804ba27d6f3cecd546d4b3b929c15399785fe5c5ab31d38280f6496e435fd16: Status 404 returned error can't find the container with id 0804ba27d6f3cecd546d4b3b929c15399785fe5c5ab31d38280f6496e435fd16
Sep 17 01:53:55.833584 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.833554    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:53:55.833663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.833619    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:53:55.833663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:55.833649    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:53:56.697152 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:56.697114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" event={"ID":"7901ff35-db82-4476-8a59-acd18a605682","Type":"ContainerStarted","Data":"e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f"}
Sep 17 01:53:56.697649 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:56.697178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" event={"ID":"7901ff35-db82-4476-8a59-acd18a605682","Type":"ContainerStarted","Data":"0804ba27d6f3cecd546d4b3b929c15399785fe5c5ab31d38280f6496e435fd16"}
Sep 17 01:53:56.715326 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:56.715271    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" podStartSLOduration=1.715256552 podStartE2EDuration="1.715256552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:53:56.713582606 +0000 UTC m=+1915.384098695" watchObservedRunningTime="2026-09-17 01:53:56.715256552 +0000 UTC m=+1915.385772663"
Sep 17 01:53:57.556341 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:57.556309    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:57.561452 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:57.561426    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:57.701695 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:57.701663    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:53:57.702632 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:53:57.702612    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:54:17.809641 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:17.809607    2578 generic.go:356] "Generic (PLEG): container finished" podID="7901ff35-db82-4476-8a59-acd18a605682" containerID="e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f" exitCode=0
Sep 17 01:54:17.810057 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:17.809679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" event={"ID":"7901ff35-db82-4476-8a59-acd18a605682","Type":"ContainerDied","Data":"e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f"}
Sep 17 01:54:17.810102 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:17.810056    2578 scope.go:117] "RemoveContainer" containerID="e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f"
Sep 17 01:54:17.811140 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:17.811113    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:54:17.811233 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:17.811189    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:54:17.811233 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:17.811219    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:54:18.816641 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:18.816605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" event={"ID":"7901ff35-db82-4476-8a59-acd18a605682","Type":"ContainerStarted","Data":"85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8"}
Sep 17 01:54:19.556199 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:19.556162    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:54:19.556379 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:19.556355    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 17 01:54:19.556433 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:19.556413    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 17 01:54:20.556705 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:20.556669    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 17 01:54:20.557090 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:20.556735    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 17 01:54:21.557280 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:21.557234    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 17 01:54:21.557801 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:21.557303    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 17 01:54:22.556604 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:22.556566    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 17 01:54:22.556786 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:22.556626    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 17 01:54:23.557233 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:23.557201    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 17 01:54:23.557623 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:23.557256    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 17 01:54:24.560669 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:24.560639    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:54:24.561094 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:24.561047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:54:24.561636 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:24.561617    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:54:38.866697 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.866661    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"]
Sep 17 01:54:38.870677 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.870653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:38.874137 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.874115    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-647268\"/\"kube-root-ca.crt\""
Sep 17 01:54:38.874444 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.874113    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-647268\"/\"ipp-route-647268-kserve-self-signed-certs\""
Sep 17 01:54:38.874683 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.874134    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-647268\"/\"openshift-service-ca.crt\""
Sep 17 01:54:38.874778 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.874196    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-647268\"/\"default-dockercfg-x52n8\""
Sep 17 01:54:38.877253 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.877208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"]
Sep 17 01:54:38.993936 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.993895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwptx\" (UniqueName: \"kubernetes.io/projected/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-kube-api-access-dwptx\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:38.994134 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.993944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-dshm\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:38.994134 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.994030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tmp-dir\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:38.994134 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.994129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tls-certs\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:38.994288 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.994156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-home\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:38.994288 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:38.994250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-model-cache\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.095009 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.094974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tls-certs\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.095009 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.095009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-home\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.095254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.095045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-model-cache\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.095254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.095066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwptx\" (UniqueName: \"kubernetes.io/projected/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-kube-api-access-dwptx\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.095254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.095098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-dshm\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.095254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.095139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tmp-dir\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.095466 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.095442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-model-cache\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.095530 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.095473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-home\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.095530 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.095524    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tmp-dir\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.097329 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.097306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-dshm\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.097662 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.097643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tls-certs\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.102955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.102932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwptx\" (UniqueName: \"kubernetes.io/projected/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-kube-api-access-dwptx\") pod \"ipp-route-647268-kserve-84cfd8668c-48n69\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") " pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.182950 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.182853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:39.455706 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.455685    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"]
Sep 17 01:54:39.457656 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:54:39.457624    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9bcf9e4_eb3d_4dee_9992_16590bd5d9e0.slice/crio-b42b731adadf0fd42e8182497a84a67aa4124800bca63e171d9c912a45705ec6 WatchSource:0}: Error finding container b42b731adadf0fd42e8182497a84a67aa4124800bca63e171d9c912a45705ec6: Status 404 returned error can't find the container with id b42b731adadf0fd42e8182497a84a67aa4124800bca63e171d9c912a45705ec6
Sep 17 01:54:39.922027 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.921990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69" event={"ID":"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0","Type":"ContainerStarted","Data":"9a08c379c3a43f188cc81296651427561cecfebc8d9c4159fe8070484e97edbf"}
Sep 17 01:54:39.922027 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.922037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69" event={"ID":"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0","Type":"ContainerStarted","Data":"b42b731adadf0fd42e8182497a84a67aa4124800bca63e171d9c912a45705ec6"}
Sep 17 01:54:39.939776 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:39.939730    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69" podStartSLOduration=1.939716397 podStartE2EDuration="1.939716397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:54:39.93752484 +0000 UTC m=+1958.608040921" watchObservedRunningTime="2026-09-17 01:54:39.939716397 +0000 UTC m=+1958.610232485"
Sep 17 01:54:49.183574 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:49.183518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:49.184177 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:49.183588    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:49.195611 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:49.195586    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:54:49.980078 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:54:49.980050    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:55:09.975128 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:09.975052    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms_38380747-222c-4dbe-a1d4-5bee2d1761f8/payload-processing/0.log"
Sep 17 01:55:11.020967 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.020915    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"]
Sep 17 01:55:11.024968 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.024944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.027091 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.027070    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-cb9976-openshift-default-dockercfg-g6k6x\""
Sep 17 01:55:11.036007 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.035984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"]
Sep 17 01:55:11.104187 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.104155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-certs\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.104377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.104192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-podinfo\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.104377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.104253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-credential-socket\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.104377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.104306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-data\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.104377 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.104353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-envoy\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.104684 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.104423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-token\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.104684 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.104445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-socket\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.104684 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.104477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d1ba50-997d-48d3-9208-7d7078e24bd9-istiod-ca-cert\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.104684 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.104511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgvlr\" (UniqueName: \"kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-kube-api-access-qgvlr\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.205816 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.205775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-envoy\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.205994 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.205867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-token\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.205994 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.205909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-socket\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.205994 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.205957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d1ba50-997d-48d3-9208-7d7078e24bd9-istiod-ca-cert\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.205994 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.205987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgvlr\" (UniqueName: \"kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-kube-api-access-qgvlr\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.206211 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.206042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-certs\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.206269 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.206219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-podinfo\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.206324 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.206282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-credential-socket\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.206378 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.206350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-data\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.206428 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.206385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-certs\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.206507 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.206483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-socket\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.206756 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.206727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-credential-socket\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.206756 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.206746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-data\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.206944 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.206800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d1ba50-997d-48d3-9208-7d7078e24bd9-istiod-ca-cert\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.208244 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.208220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-envoy\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.208592 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.208576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-podinfo\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.227085 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.227050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-token\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.228797 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.228772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgvlr\" (UniqueName: \"kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-kube-api-access-qgvlr\") pod \"e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") " pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.337299 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.337270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:11.623404 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.623379    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"]
Sep 17 01:55:11.625783 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:55:11.625740    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d1ba50_997d_48d3_9208_7d7078e24bd9.slice/crio-5c518ccfb77bcdd8388a9a3416b6f382144ea7c2de0dd9fe0b0cc06b8b6d6615 WatchSource:0}: Error finding container 5c518ccfb77bcdd8388a9a3416b6f382144ea7c2de0dd9fe0b0cc06b8b6d6615: Status 404 returned error can't find the container with id 5c518ccfb77bcdd8388a9a3416b6f382144ea7c2de0dd9fe0b0cc06b8b6d6615
Sep 17 01:55:11.628098 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.628068    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:55:11.628433 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.628128    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:55:11.628433 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:11.628181    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:55:12.079062 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:12.079025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" event={"ID":"44d1ba50-997d-48d3-9208-7d7078e24bd9","Type":"ContainerStarted","Data":"f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b"}
Sep 17 01:55:12.079475 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:12.079082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" event={"ID":"44d1ba50-997d-48d3-9208-7d7078e24bd9","Type":"ContainerStarted","Data":"5c518ccfb77bcdd8388a9a3416b6f382144ea7c2de0dd9fe0b0cc06b8b6d6615"}
Sep 17 01:55:12.099268 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:12.099207    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" podStartSLOduration=1.099187696 podStartE2EDuration="1.099187696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:12.098620364 +0000 UTC m=+1990.769136453" watchObservedRunningTime="2026-09-17 01:55:12.099187696 +0000 UTC m=+1990.769703787"
Sep 17 01:55:12.338240 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:12.338155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:12.343947 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:12.343921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:13.083893 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:13.083858    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:13.084847 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:13.084830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:17.517414 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.517384    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"]
Sep 17 01:55:17.521278 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.521254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:17.523660 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.523634    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-cb9976\""
Sep 17 01:55:17.523777 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.523634    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-cb9976-dockercfg-hnv7c\""
Sep 17 01:55:17.531751 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.531728    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"]
Sep 17 01:55:17.580276 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.570571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846b0ac0-49cf-46a1-9a6a-d0bf10008503-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7\" (UID: \"846b0ac0-49cf-46a1-9a6a-d0bf10008503\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:17.580276 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.570696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92866\" (UniqueName: \"kubernetes.io/projected/846b0ac0-49cf-46a1-9a6a-d0bf10008503-kube-api-access-92866\") pod \"payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7\" (UID: \"846b0ac0-49cf-46a1-9a6a-d0bf10008503\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:17.672030 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.671996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846b0ac0-49cf-46a1-9a6a-d0bf10008503-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7\" (UID: \"846b0ac0-49cf-46a1-9a6a-d0bf10008503\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:17.672202 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.672058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92866\" (UniqueName: \"kubernetes.io/projected/846b0ac0-49cf-46a1-9a6a-d0bf10008503-kube-api-access-92866\") pod \"payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7\" (UID: \"846b0ac0-49cf-46a1-9a6a-d0bf10008503\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:17.672701 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.672682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846b0ac0-49cf-46a1-9a6a-d0bf10008503-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7\" (UID: \"846b0ac0-49cf-46a1-9a6a-d0bf10008503\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:17.679079 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.679055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92866\" (UniqueName: \"kubernetes.io/projected/846b0ac0-49cf-46a1-9a6a-d0bf10008503-kube-api-access-92866\") pod \"payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7\" (UID: \"846b0ac0-49cf-46a1-9a6a-d0bf10008503\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:17.834500 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:17.834470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:18.107353 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:18.107316    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"]
Sep 17 01:55:18.111329 ip-10-0-130-87 kubenswrapper[2578]: W0917 01:55:18.111303    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846b0ac0_49cf_46a1_9a6a_d0bf10008503.slice/crio-4ad0fd65a4c9148cfc32cdd9294ea3c245b81dcbca5f567f941013fb39b5817f WatchSource:0}: Error finding container 4ad0fd65a4c9148cfc32cdd9294ea3c245b81dcbca5f567f941013fb39b5817f: Status 404 returned error can't find the container with id 4ad0fd65a4c9148cfc32cdd9294ea3c245b81dcbca5f567f941013fb39b5817f
Sep 17 01:55:19.117942 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:19.117908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7" event={"ID":"846b0ac0-49cf-46a1-9a6a-d0bf10008503","Type":"ContainerStarted","Data":"828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03"}
Sep 17 01:55:19.118345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:19.117958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7" event={"ID":"846b0ac0-49cf-46a1-9a6a-d0bf10008503","Type":"ContainerStarted","Data":"4ad0fd65a4c9148cfc32cdd9294ea3c245b81dcbca5f567f941013fb39b5817f"}
Sep 17 01:55:19.118345 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:19.118051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:19.136209 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:19.136161    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7" podStartSLOduration=2.136137348 podStartE2EDuration="2.136137348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:19.133604494 +0000 UTC m=+1997.804120584" watchObservedRunningTime="2026-09-17 01:55:19.136137348 +0000 UTC m=+1997.806653437"
Sep 17 01:55:30.126334 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:30.126297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:33.201104 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:33.201072    2578 generic.go:356] "Generic (PLEG): container finished" podID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerID="f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b" exitCode=0
Sep 17 01:55:33.201504 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:33.201149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" event={"ID":"44d1ba50-997d-48d3-9208-7d7078e24bd9","Type":"ContainerDied","Data":"f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b"}
Sep 17 01:55:33.201638 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:33.201623    2578 scope.go:117] "RemoveContainer" containerID="f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b"
Sep 17 01:55:33.202751 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:33.202727    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:55:33.202813 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:33.202790    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:55:33.202851 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:33.202819    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 01:55:34.208656 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:34.208626    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" event={"ID":"44d1ba50-997d-48d3-9208-7d7078e24bd9","Type":"ContainerStarted","Data":"1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c"}
Sep 17 01:55:34.338387 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:34.338351    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:34.338614 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:34.338591    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 17 01:55:34.338679 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:34.338642    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 17 01:55:35.338281 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:35.338244    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 17 01:55:35.338679 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:35.338306    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 17 01:55:36.337935 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:36.337896    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 17 01:55:36.338115 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:36.337959    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 17 01:55:37.337857 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:37.337822    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 17 01:55:37.338254 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:37.337882    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 17 01:55:38.338041 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:38.338003    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 17 01:55:38.338440 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:38.338064    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 17 01:55:39.385419 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:39.385378    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:39.385913 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:39.385891    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:39.387865 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:39.387842    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:45.962574 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:45.962523    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"]
Sep 17 01:55:45.963042 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:45.962793    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7" podUID="846b0ac0-49cf-46a1-9a6a-d0bf10008503" containerName="payload-processing" containerID="cri-o://828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03" gracePeriod=30
Sep 17 01:55:46.208714 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.208691    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:46.273392 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.273303    2578 generic.go:356] "Generic (PLEG): container finished" podID="846b0ac0-49cf-46a1-9a6a-d0bf10008503" containerID="828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03" exitCode=0
Sep 17 01:55:46.273392 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.273343    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"
Sep 17 01:55:46.273392 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.273356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7" event={"ID":"846b0ac0-49cf-46a1-9a6a-d0bf10008503","Type":"ContainerDied","Data":"828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03"}
Sep 17 01:55:46.273656 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.273407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7" event={"ID":"846b0ac0-49cf-46a1-9a6a-d0bf10008503","Type":"ContainerDied","Data":"4ad0fd65a4c9148cfc32cdd9294ea3c245b81dcbca5f567f941013fb39b5817f"}
Sep 17 01:55:46.273656 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.273437    2578 scope.go:117] "RemoveContainer" containerID="828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03"
Sep 17 01:55:46.278042 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.277997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92866\" (UniqueName: \"kubernetes.io/projected/846b0ac0-49cf-46a1-9a6a-d0bf10008503-kube-api-access-92866\") pod \"846b0ac0-49cf-46a1-9a6a-d0bf10008503\" (UID: \"846b0ac0-49cf-46a1-9a6a-d0bf10008503\") "
Sep 17 01:55:46.278157 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.278120    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846b0ac0-49cf-46a1-9a6a-d0bf10008503-plugins-config-volume\") pod \"846b0ac0-49cf-46a1-9a6a-d0bf10008503\" (UID: \"846b0ac0-49cf-46a1-9a6a-d0bf10008503\") "
Sep 17 01:55:46.278491 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.278462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846b0ac0-49cf-46a1-9a6a-d0bf10008503-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "846b0ac0-49cf-46a1-9a6a-d0bf10008503" (UID: "846b0ac0-49cf-46a1-9a6a-d0bf10008503"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:55:46.280518 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.280499    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846b0ac0-49cf-46a1-9a6a-d0bf10008503-kube-api-access-92866" (OuterVolumeSpecName: "kube-api-access-92866") pod "846b0ac0-49cf-46a1-9a6a-d0bf10008503" (UID: "846b0ac0-49cf-46a1-9a6a-d0bf10008503"). InnerVolumeSpecName "kube-api-access-92866". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:55:46.282815 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.282763    2578 scope.go:117] "RemoveContainer" containerID="828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03"
Sep 17 01:55:46.283059 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:55:46.283042    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03\": container with ID starting with 828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03 not found: ID does not exist" containerID="828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03"
Sep 17 01:55:46.283116 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.283069    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03"} err="failed to get container status \"828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03\": rpc error: code = NotFound desc = could not find container \"828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03\": container with ID starting with 828655f4aa8e2f651343cb06f2a5b6ea7703128b92ea011225b80b8edf0bcf03 not found: ID does not exist"
Sep 17 01:55:46.379331 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.379297    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846b0ac0-49cf-46a1-9a6a-d0bf10008503-plugins-config-volume\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:46.379331 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.379327    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92866\" (UniqueName: \"kubernetes.io/projected/846b0ac0-49cf-46a1-9a6a-d0bf10008503-kube-api-access-92866\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:46.594742 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.594709    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"]
Sep 17 01:55:46.600436 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:46.600413    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-cb9976-5bbbff5547-9g2v7"]
Sep 17 01:55:47.921452 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:47.921415    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="846b0ac0-49cf-46a1-9a6a-d0bf10008503" path="/var/lib/kubelet/pods/846b0ac0-49cf-46a1-9a6a-d0bf10008503/volumes"
Sep 17 01:55:52.530143 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:52.530106    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"]
Sep 17 01:55:52.530638 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:52.530380    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy" containerID="cri-o://1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c" gracePeriod=30
Sep 17 01:55:54.388307 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:54.388267    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 01:55:54.388717 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:54.388336    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 01:55:57.787740 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.787714    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:57.886753 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.886726    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-certs\") pod \"44d1ba50-997d-48d3-9208-7d7078e24bd9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") "
Sep 17 01:55:57.886943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.886796    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-podinfo\") pod \"44d1ba50-997d-48d3-9208-7d7078e24bd9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") "
Sep 17 01:55:57.886943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.886828    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-data\") pod \"44d1ba50-997d-48d3-9208-7d7078e24bd9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") "
Sep 17 01:55:57.886943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.886868    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-token\") pod \"44d1ba50-997d-48d3-9208-7d7078e24bd9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") "
Sep 17 01:55:57.886943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.886902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d1ba50-997d-48d3-9208-7d7078e24bd9-istiod-ca-cert\") pod \"44d1ba50-997d-48d3-9208-7d7078e24bd9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") "
Sep 17 01:55:57.886943 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.886934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-socket\") pod \"44d1ba50-997d-48d3-9208-7d7078e24bd9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") "
Sep 17 01:55:57.887225 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.886975    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgvlr\" (UniqueName: \"kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-kube-api-access-qgvlr\") pod \"44d1ba50-997d-48d3-9208-7d7078e24bd9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") "
Sep 17 01:55:57.887225 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.886992    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "44d1ba50-997d-48d3-9208-7d7078e24bd9" (UID: "44d1ba50-997d-48d3-9208-7d7078e24bd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:55:57.887225 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-credential-socket\") pod \"44d1ba50-997d-48d3-9208-7d7078e24bd9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") "
Sep 17 01:55:57.887225 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887066    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-envoy\") pod \"44d1ba50-997d-48d3-9208-7d7078e24bd9\" (UID: \"44d1ba50-997d-48d3-9208-7d7078e24bd9\") "
Sep 17 01:55:57.887225 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887130    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "44d1ba50-997d-48d3-9208-7d7078e24bd9" (UID: "44d1ba50-997d-48d3-9208-7d7078e24bd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:55:57.887225 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "44d1ba50-997d-48d3-9208-7d7078e24bd9" (UID: "44d1ba50-997d-48d3-9208-7d7078e24bd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:55:57.887527 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887285    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d1ba50-997d-48d3-9208-7d7078e24bd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "44d1ba50-997d-48d3-9208-7d7078e24bd9" (UID: "44d1ba50-997d-48d3-9208-7d7078e24bd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:55:57.887527 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887305    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "44d1ba50-997d-48d3-9208-7d7078e24bd9" (UID: "44d1ba50-997d-48d3-9208-7d7078e24bd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:55:57.887672 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887530    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:57.887672 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887584    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-data\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:57.887672 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887597    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d1ba50-997d-48d3-9208-7d7078e24bd9-istiod-ca-cert\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:57.887672 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887609    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-workload-socket\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:57.887672 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.887621    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-credential-socket\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:57.889305 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.889278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "44d1ba50-997d-48d3-9208-7d7078e24bd9" (UID: "44d1ba50-997d-48d3-9208-7d7078e24bd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 01:55:57.889413 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.889303    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "44d1ba50-997d-48d3-9208-7d7078e24bd9" (UID: "44d1ba50-997d-48d3-9208-7d7078e24bd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:55:57.889413 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.889333    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "44d1ba50-997d-48d3-9208-7d7078e24bd9" (UID: "44d1ba50-997d-48d3-9208-7d7078e24bd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:55:57.889413 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.889341    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-kube-api-access-qgvlr" (OuterVolumeSpecName: "kube-api-access-qgvlr") pod "44d1ba50-997d-48d3-9208-7d7078e24bd9" (UID: "44d1ba50-997d-48d3-9208-7d7078e24bd9"). InnerVolumeSpecName "kube-api-access-qgvlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:55:57.995877 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.991074    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgvlr\" (UniqueName: \"kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-kube-api-access-qgvlr\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:57.995877 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.991108    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-envoy\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:57.995877 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.991129    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-podinfo\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:57.995877 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:57.991143    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d1ba50-997d-48d3-9208-7d7078e24bd9-istio-token\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:55:58.337198 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.337163    2578 generic.go:356] "Generic (PLEG): container finished" podID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerID="1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c" exitCode=0
Sep 17 01:55:58.337360 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.337220    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"
Sep 17 01:55:58.337360 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.337244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" event={"ID":"44d1ba50-997d-48d3-9208-7d7078e24bd9","Type":"ContainerDied","Data":"1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c"}
Sep 17 01:55:58.337360 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.337319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9" event={"ID":"44d1ba50-997d-48d3-9208-7d7078e24bd9","Type":"ContainerDied","Data":"5c518ccfb77bcdd8388a9a3416b6f382144ea7c2de0dd9fe0b0cc06b8b6d6615"}
Sep 17 01:55:58.337360 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.337346    2578 scope.go:117] "RemoveContainer" containerID="1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c"
Sep 17 01:55:58.345921 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.345907    2578 scope.go:117] "RemoveContainer" containerID="f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b"
Sep 17 01:55:58.354281 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.354264    2578 scope.go:117] "RemoveContainer" containerID="1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c"
Sep 17 01:55:58.354571 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:55:58.354552    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c\": container with ID starting with 1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c not found: ID does not exist" containerID="1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c"
Sep 17 01:55:58.354673 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.354578    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c"} err="failed to get container status \"1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c\": rpc error: code = NotFound desc = could not find container \"1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c\": container with ID starting with 1a4e71bb43e630184d70cf77f7e7a06a1511d166ebe3abef1077e434432e843c not found: ID does not exist"
Sep 17 01:55:58.354673 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.354594    2578 scope.go:117] "RemoveContainer" containerID="f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b"
Sep 17 01:55:58.354837 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:55:58.354823    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b\": container with ID starting with f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b not found: ID does not exist" containerID="f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b"
Sep 17 01:55:58.354882 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.354839    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b"} err="failed to get container status \"f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b\": rpc error: code = NotFound desc = could not find container \"f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b\": container with ID starting with f9003c4c247fdb79e049cf4979105b6f6f6032cdf80de06be72eba68fd25404b not found: ID does not exist"
Sep 17 01:55:58.355624 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.355606    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"]
Sep 17 01:55:58.362030 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:58.362009    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cb9976-openshift-default-d76b45b7b-wsxs9"]
Sep 17 01:55:59.923455 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:55:59.923423    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" path="/var/lib/kubelet/pods/44d1ba50-997d-48d3-9208-7d7078e24bd9/volumes"
Sep 17 01:56:11.741051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:11.741018    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"]
Sep 17 01:56:11.741579 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:11.741260    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms" podUID="38380747-222c-4dbe-a1d4-5bee2d1761f8" containerName="payload-processing" containerID="cri-o://ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0" gracePeriod=30
Sep 17 01:56:11.744725 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:11.744679    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"]
Sep 17 01:56:11.744923 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:11.744890    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69" podUID="abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38" containerName="payload-pre-processing" containerID="cri-o://d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c" gracePeriod=30
Sep 17 01:56:12.027471 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.027447    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:56:12.030589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.030568    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:56:12.131270 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.131240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-plugins-config-volume\") pod \"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38\" (UID: \"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38\") "
Sep 17 01:56:12.131476 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.131286    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fj4t\" (UniqueName: \"kubernetes.io/projected/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-kube-api-access-6fj4t\") pod \"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38\" (UID: \"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38\") "
Sep 17 01:56:12.131476 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.131372    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38380747-222c-4dbe-a1d4-5bee2d1761f8-plugins-config-volume\") pod \"38380747-222c-4dbe-a1d4-5bee2d1761f8\" (UID: \"38380747-222c-4dbe-a1d4-5bee2d1761f8\") "
Sep 17 01:56:12.131476 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.131409    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54nfq\" (UniqueName: \"kubernetes.io/projected/38380747-222c-4dbe-a1d4-5bee2d1761f8-kube-api-access-54nfq\") pod \"38380747-222c-4dbe-a1d4-5bee2d1761f8\" (UID: \"38380747-222c-4dbe-a1d4-5bee2d1761f8\") "
Sep 17 01:56:12.131770 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.131742    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38" (UID: "abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:56:12.131860 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.131839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38380747-222c-4dbe-a1d4-5bee2d1761f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38380747-222c-4dbe-a1d4-5bee2d1761f8" (UID: "38380747-222c-4dbe-a1d4-5bee2d1761f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:56:12.133602 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.133577    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38380747-222c-4dbe-a1d4-5bee2d1761f8-kube-api-access-54nfq" (OuterVolumeSpecName: "kube-api-access-54nfq") pod "38380747-222c-4dbe-a1d4-5bee2d1761f8" (UID: "38380747-222c-4dbe-a1d4-5bee2d1761f8"). InnerVolumeSpecName "kube-api-access-54nfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:56:12.133674 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.133611    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-kube-api-access-6fj4t" (OuterVolumeSpecName: "kube-api-access-6fj4t") pod "abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38" (UID: "abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38"). InnerVolumeSpecName "kube-api-access-6fj4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:56:12.232798 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.232761    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38380747-222c-4dbe-a1d4-5bee2d1761f8-plugins-config-volume\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:12.232798 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.232791    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54nfq\" (UniqueName: \"kubernetes.io/projected/38380747-222c-4dbe-a1d4-5bee2d1761f8-kube-api-access-54nfq\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:12.232798 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.232803    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-plugins-config-volume\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:12.233025 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.232815    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fj4t\" (UniqueName: \"kubernetes.io/projected/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38-kube-api-access-6fj4t\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:12.406406 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.406374    2578 generic.go:356] "Generic (PLEG): container finished" podID="abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38" containerID="d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c" exitCode=0
Sep 17 01:56:12.406663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.406413    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"
Sep 17 01:56:12.406663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.406427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69" event={"ID":"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38","Type":"ContainerDied","Data":"d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c"}
Sep 17 01:56:12.406663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.406488    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69" event={"ID":"abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38","Type":"ContainerDied","Data":"ea19b084c3c7b0b77352ee40cad4830fe76717d5a9981f3d92072690c8095bc5"}
Sep 17 01:56:12.406663 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.406517    2578 scope.go:117] "RemoveContainer" containerID="d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c"
Sep 17 01:56:12.407873 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.407849    2578 generic.go:356] "Generic (PLEG): container finished" podID="38380747-222c-4dbe-a1d4-5bee2d1761f8" containerID="ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0" exitCode=0
Sep 17 01:56:12.407975 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.407894    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"
Sep 17 01:56:12.407975 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.407903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms" event={"ID":"38380747-222c-4dbe-a1d4-5bee2d1761f8","Type":"ContainerDied","Data":"ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0"}
Sep 17 01:56:12.407975 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.407937    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms" event={"ID":"38380747-222c-4dbe-a1d4-5bee2d1761f8","Type":"ContainerDied","Data":"d0dfbdcdc348e46a1f146b330a21bf66d4d1a640bf11a4fc97c4d6c775edd453"}
Sep 17 01:56:12.415392 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.415367    2578 scope.go:117] "RemoveContainer" containerID="d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c"
Sep 17 01:56:12.415729 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:56:12.415708    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c\": container with ID starting with d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c not found: ID does not exist" containerID="d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c"
Sep 17 01:56:12.415830 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.415741    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c"} err="failed to get container status \"d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c\": rpc error: code = NotFound desc = could not find container \"d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c\": container with ID starting with d92b3be7a57c30b0e4c2f00a69d278975bf6e8afedc0da293a9419ea41cf6d3c not found: ID does not exist"
Sep 17 01:56:12.415830 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.415762    2578 scope.go:117] "RemoveContainer" containerID="ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0"
Sep 17 01:56:12.426981 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.426964    2578 scope.go:117] "RemoveContainer" containerID="ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0"
Sep 17 01:56:12.427243 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:56:12.427224    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0\": container with ID starting with ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0 not found: ID does not exist" containerID="ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0"
Sep 17 01:56:12.427303 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.427253    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0"} err="failed to get container status \"ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0\": rpc error: code = NotFound desc = could not find container \"ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0\": container with ID starting with ef02bda23d91487b745891ee1bcace96f62a8a6d02dc1fb58f245edb3c5b98b0 not found: ID does not exist"
Sep 17 01:56:12.433608 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.433585    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"]
Sep 17 01:56:12.437597 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.437577    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-647268-5b665fc9f9-blrms"]
Sep 17 01:56:12.449918 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.449894    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"]
Sep 17 01:56:12.455011 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:12.454991    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-647268-54c7b8d958-69b69"]
Sep 17 01:56:13.920994 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:13.920960    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38380747-222c-4dbe-a1d4-5bee2d1761f8" path="/var/lib/kubelet/pods/38380747-222c-4dbe-a1d4-5bee2d1761f8/volumes"
Sep 17 01:56:13.921361 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:13.921332    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38" path="/var/lib/kubelet/pods/abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38/volumes"
Sep 17 01:56:17.239921 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:17.239878    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"]
Sep 17 01:56:32.259589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.259462    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69" podUID="f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" containerName="main" containerID="cri-o://9a08c379c3a43f188cc81296651427561cecfebc8d9c4159fe8070484e97edbf" gracePeriod=45
Sep 17 01:56:32.508295 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.508265    2578 generic.go:356] "Generic (PLEG): container finished" podID="f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" containerID="9a08c379c3a43f188cc81296651427561cecfebc8d9c4159fe8070484e97edbf" exitCode=0
Sep 17 01:56:32.508427 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.508315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69" event={"ID":"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0","Type":"ContainerDied","Data":"9a08c379c3a43f188cc81296651427561cecfebc8d9c4159fe8070484e97edbf"}
Sep 17 01:56:32.508427 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.508343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69" event={"ID":"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0","Type":"ContainerDied","Data":"b42b731adadf0fd42e8182497a84a67aa4124800bca63e171d9c912a45705ec6"}
Sep 17 01:56:32.508427 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.508352    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b42b731adadf0fd42e8182497a84a67aa4124800bca63e171d9c912a45705ec6"
Sep 17 01:56:32.517258 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.517207    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:56:32.618955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.618921    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tls-certs\") pod \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") "
Sep 17 01:56:32.618955 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.618961    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-dshm\") pod \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") "
Sep 17 01:56:32.619176 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.619001    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-home\") pod \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") "
Sep 17 01:56:32.619176 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.619021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwptx\" (UniqueName: \"kubernetes.io/projected/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-kube-api-access-dwptx\") pod \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") "
Sep 17 01:56:32.619176 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.619046    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-model-cache\") pod \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") "
Sep 17 01:56:32.619176 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.619129    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tmp-dir\") pod \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\" (UID: \"f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0\") "
Sep 17 01:56:32.619394 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.619260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-home" (OuterVolumeSpecName: "home") pod "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" (UID: "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:56:32.619472 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.619405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" (UID: "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:56:32.619523 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.619474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" (UID: "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:56:32.619598 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.619579    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:32.619638 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.619598    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:32.621284 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.621261    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-dshm" (OuterVolumeSpecName: "dshm") pod "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" (UID: "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:56:32.621382 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.621300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" (UID: "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:56:32.621382 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.621320    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-kube-api-access-dwptx" (OuterVolumeSpecName: "kube-api-access-dwptx") pod "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" (UID: "f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0"). InnerVolumeSpecName "kube-api-access-dwptx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:56:32.721102 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.721063    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:32.721102 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.721096    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:32.721102 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.721107    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:32.721346 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:32.721116    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwptx\" (UniqueName: \"kubernetes.io/projected/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0-kube-api-access-dwptx\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:33.513718 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:33.513690    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"
Sep 17 01:56:33.534896 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:33.534869    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"]
Sep 17 01:56:33.538723 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:33.538699    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-647268/ipp-route-647268-kserve-84cfd8668c-48n69"]
Sep 17 01:56:33.921556 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:33.921514    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" path="/var/lib/kubelet/pods/f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0/volumes"
Sep 17 01:56:48.493976 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:48.493940    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"]
Sep 17 01:56:48.494497 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:48.494218    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy" containerID="cri-o://38af956206293518f1ec86583ac9bb66d2c71704e5237b95280f7cea99483a23" gracePeriod=30
Sep 17 01:56:53.618982 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.618950    2578 generic.go:356] "Generic (PLEG): container finished" podID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerID="38af956206293518f1ec86583ac9bb66d2c71704e5237b95280f7cea99483a23" exitCode=0
Sep 17 01:56:53.619423 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.618999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" event={"ID":"2a457cf9-d9c1-489d-b589-9afb8edeb52c","Type":"ContainerDied","Data":"38af956206293518f1ec86583ac9bb66d2c71704e5237b95280f7cea99483a23"}
Sep 17 01:56:53.619423 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.619048    2578 scope.go:117] "RemoveContainer" containerID="fbd3646ad744d4c7172f3dae241314accef459fcf9ba0b3ea38195a8e7eb1a94"
Sep 17 01:56:53.746064 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.746037    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:56:53.934870 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.934844    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-socket\") pod \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") "
Sep 17 01:56:53.935055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.934899    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-certs\") pod \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") "
Sep 17 01:56:53.935055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.934945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-podinfo\") pod \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") "
Sep 17 01:56:53.935055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.934981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-credential-socket\") pod \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") "
Sep 17 01:56:53.935055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-data\") pod \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") "
Sep 17 01:56:53.935055 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935046    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-envoy\") pod \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") "
Sep 17 01:56:53.935337 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935101    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pftkt\" (UniqueName: \"kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-kube-api-access-pftkt\") pod \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") "
Sep 17 01:56:53.935337 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935134    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istiod-ca-cert\") pod \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") "
Sep 17 01:56:53.935337 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935168    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-token\") pod \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\" (UID: \"2a457cf9-d9c1-489d-b589-9afb8edeb52c\") "
Sep 17 01:56:53.935337 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935095    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a457cf9-d9c1-489d-b589-9afb8edeb52c" (UID: "2a457cf9-d9c1-489d-b589-9afb8edeb52c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:56:53.935337 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a457cf9-d9c1-489d-b589-9afb8edeb52c" (UID: "2a457cf9-d9c1-489d-b589-9afb8edeb52c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:56:53.935337 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a457cf9-d9c1-489d-b589-9afb8edeb52c" (UID: "2a457cf9-d9c1-489d-b589-9afb8edeb52c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:56:53.935612 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935382    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a457cf9-d9c1-489d-b589-9afb8edeb52c" (UID: "2a457cf9-d9c1-489d-b589-9afb8edeb52c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:56:53.935612 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935598    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-credential-socket\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:53.935688 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935620    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-data\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:53.935688 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a457cf9-d9c1-489d-b589-9afb8edeb52c" (UID: "2a457cf9-d9c1-489d-b589-9afb8edeb52c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:56:53.935688 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935634    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-socket\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:53.935688 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.935647    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-workload-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:53.937310 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.937281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-kube-api-access-pftkt" (OuterVolumeSpecName: "kube-api-access-pftkt") pod "2a457cf9-d9c1-489d-b589-9afb8edeb52c" (UID: "2a457cf9-d9c1-489d-b589-9afb8edeb52c"). InnerVolumeSpecName "kube-api-access-pftkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:56:53.937477 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.937462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a457cf9-d9c1-489d-b589-9afb8edeb52c" (UID: "2a457cf9-d9c1-489d-b589-9afb8edeb52c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:56:53.937662 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.937646    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a457cf9-d9c1-489d-b589-9afb8edeb52c" (UID: "2a457cf9-d9c1-489d-b589-9afb8edeb52c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 01:56:53.937718 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:53.937674    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a457cf9-d9c1-489d-b589-9afb8edeb52c" (UID: "2a457cf9-d9c1-489d-b589-9afb8edeb52c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:56:54.036589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.036558    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-envoy\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:54.036589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.036587    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pftkt\" (UniqueName: \"kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-kube-api-access-pftkt\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:54.036589 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.036598    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istiod-ca-cert\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:54.036903 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.036607    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-token\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:54.036903 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.036615    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a457cf9-d9c1-489d-b589-9afb8edeb52c-istio-podinfo\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:56:54.625490 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.625449    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" event={"ID":"2a457cf9-d9c1-489d-b589-9afb8edeb52c","Type":"ContainerDied","Data":"5f48fb74e88f22a13124103b67af17eaa2f36e3b3b6d6677991aa5019cf45395"}
Sep 17 01:56:54.625922 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.625511    2578 scope.go:117] "RemoveContainer" containerID="38af956206293518f1ec86583ac9bb66d2c71704e5237b95280f7cea99483a23"
Sep 17 01:56:54.625922 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.625466    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"
Sep 17 01:56:54.646435 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.646409    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"]
Sep 17 01:56:54.649807 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.649778    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9"]
Sep 17 01:56:54.687686 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.687652    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 01:56:54.687795 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:54.687721    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-647268-openshift-default-68988998dd-s9jn9" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 01:56:55.920441 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:56:55.920406    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" path="/var/lib/kubelet/pods/2a457cf9-d9c1-489d-b589-9afb8edeb52c/volumes"
Sep 17 01:57:05.885909 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:05.885875    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"]
Sep 17 01:57:05.886399 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:05.886139    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy" containerID="cri-o://85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8" gracePeriod=30
Sep 17 01:57:09.563250 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:09.563207    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 01:57:09.563645 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:09.563274    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 01:57:11.143573 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.143527    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:57:11.187554 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187457    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-certs\") pod \"7901ff35-db82-4476-8a59-acd18a605682\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") "
Sep 17 01:57:11.187554 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187491    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7901ff35-db82-4476-8a59-acd18a605682-istiod-ca-cert\") pod \"7901ff35-db82-4476-8a59-acd18a605682\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") "
Sep 17 01:57:11.187554 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-data\") pod \"7901ff35-db82-4476-8a59-acd18a605682\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") "
Sep 17 01:57:11.187797 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187639    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-istio-token\") pod \"7901ff35-db82-4476-8a59-acd18a605682\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") "
Sep 17 01:57:11.187797 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187673    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-envoy\") pod \"7901ff35-db82-4476-8a59-acd18a605682\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") "
Sep 17 01:57:11.187797 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zk2h\" (UniqueName: \"kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-kube-api-access-8zk2h\") pod \"7901ff35-db82-4476-8a59-acd18a605682\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") "
Sep 17 01:57:11.187797 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187777    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7901ff35-db82-4476-8a59-acd18a605682-istio-podinfo\") pod \"7901ff35-db82-4476-8a59-acd18a605682\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") "
Sep 17 01:57:11.188051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187806    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7901ff35-db82-4476-8a59-acd18a605682" (UID: "7901ff35-db82-4476-8a59-acd18a605682"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:57:11.188051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187833    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-socket\") pod \"7901ff35-db82-4476-8a59-acd18a605682\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") "
Sep 17 01:57:11.188051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-credential-socket\") pod \"7901ff35-db82-4476-8a59-acd18a605682\" (UID: \"7901ff35-db82-4476-8a59-acd18a605682\") "
Sep 17 01:57:11.188051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-data" (OuterVolumeSpecName: "istio-data") pod "7901ff35-db82-4476-8a59-acd18a605682" (UID: "7901ff35-db82-4476-8a59-acd18a605682"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:57:11.188051 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.187910    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7901ff35-db82-4476-8a59-acd18a605682-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7901ff35-db82-4476-8a59-acd18a605682" (UID: "7901ff35-db82-4476-8a59-acd18a605682"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:57:11.188316 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.188062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7901ff35-db82-4476-8a59-acd18a605682" (UID: "7901ff35-db82-4476-8a59-acd18a605682"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:57:11.188316 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.188268    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7901ff35-db82-4476-8a59-acd18a605682" (UID: "7901ff35-db82-4476-8a59-acd18a605682"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:57:11.188420 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.188329    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-socket\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:57:11.188420 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.188348    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-workload-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:57:11.188420 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.188364    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7901ff35-db82-4476-8a59-acd18a605682-istiod-ca-cert\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:57:11.188420 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.188377    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-data\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:57:11.189978 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.189955    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7901ff35-db82-4476-8a59-acd18a605682-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7901ff35-db82-4476-8a59-acd18a605682" (UID: "7901ff35-db82-4476-8a59-acd18a605682"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 01:57:11.190353 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.190327    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-istio-token" (OuterVolumeSpecName: "istio-token") pod "7901ff35-db82-4476-8a59-acd18a605682" (UID: "7901ff35-db82-4476-8a59-acd18a605682"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:57:11.190459 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.190392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7901ff35-db82-4476-8a59-acd18a605682" (UID: "7901ff35-db82-4476-8a59-acd18a605682"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:57:11.190459 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.190417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-kube-api-access-8zk2h" (OuterVolumeSpecName: "kube-api-access-8zk2h") pod "7901ff35-db82-4476-8a59-acd18a605682" (UID: "7901ff35-db82-4476-8a59-acd18a605682"). InnerVolumeSpecName "kube-api-access-8zk2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:57:11.289181 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.289148    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7901ff35-db82-4476-8a59-acd18a605682-istio-podinfo\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:57:11.289181 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.289176    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-credential-socket\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:57:11.289181 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.289187    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-istio-token\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:57:11.289416 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.289196    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7901ff35-db82-4476-8a59-acd18a605682-istio-envoy\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:57:11.289416 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.289203    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zk2h\" (UniqueName: \"kubernetes.io/projected/7901ff35-db82-4476-8a59-acd18a605682-kube-api-access-8zk2h\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 01:57:11.708657 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.708616    2578 generic.go:356] "Generic (PLEG): container finished" podID="7901ff35-db82-4476-8a59-acd18a605682" containerID="85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8" exitCode=0
Sep 17 01:57:11.708833 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.708662    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"
Sep 17 01:57:11.708833 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.708673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" event={"ID":"7901ff35-db82-4476-8a59-acd18a605682","Type":"ContainerDied","Data":"85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8"}
Sep 17 01:57:11.708833 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.708722    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd" event={"ID":"7901ff35-db82-4476-8a59-acd18a605682","Type":"ContainerDied","Data":"0804ba27d6f3cecd546d4b3b929c15399785fe5c5ab31d38280f6496e435fd16"}
Sep 17 01:57:11.708833 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.708746    2578 scope.go:117] "RemoveContainer" containerID="85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8"
Sep 17 01:57:11.718364 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.718347    2578 scope.go:117] "RemoveContainer" containerID="e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f"
Sep 17 01:57:11.726346 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.726327    2578 scope.go:117] "RemoveContainer" containerID="85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8"
Sep 17 01:57:11.726693 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:57:11.726666    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8\": container with ID starting with 85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8 not found: ID does not exist" containerID="85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8"
Sep 17 01:57:11.726764 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.726704    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8"} err="failed to get container status \"85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8\": rpc error: code = NotFound desc = could not find container \"85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8\": container with ID starting with 85115e614290f97d5394387286a353c2472411c9def043fea2d889a89fd5c3f8 not found: ID does not exist"
Sep 17 01:57:11.726764 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.726726    2578 scope.go:117] "RemoveContainer" containerID="e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f"
Sep 17 01:57:11.726995 ip-10-0-130-87 kubenswrapper[2578]: E0917 01:57:11.726974    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f\": container with ID starting with e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f not found: ID does not exist" containerID="e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f"
Sep 17 01:57:11.727032 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.727002    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f"} err="failed to get container status \"e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f\": rpc error: code = NotFound desc = could not find container \"e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f\": container with ID starting with e6d1953635cf215ce6eadb787bc17377bce18801f7e70740ddd335d5355d337f not found: ID does not exist"
Sep 17 01:57:11.729708 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.729689    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"]
Sep 17 01:57:11.735699 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.735664    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-29e19c-openshift-default-5c86bb444b-wqgsd"]
Sep 17 01:57:11.920636 ip-10-0-130-87 kubenswrapper[2578]: I0917 01:57:11.920603    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7901ff35-db82-4476-8a59-acd18a605682" path="/var/lib/kubelet/pods/7901ff35-db82-4476-8a59-acd18a605682/volumes"
Sep 17 02:01:02.155728 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:02.155643    2578 scope.go:117] "RemoveContainer" containerID="9a08c379c3a43f188cc81296651427561cecfebc8d9c4159fe8070484e97edbf"
Sep 17 02:01:53.840661 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.840626    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"]
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.840964    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.840975    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.840983    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846b0ac0-49cf-46a1-9a6a-d0bf10008503" containerName="payload-processing"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.840989    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="846b0ac0-49cf-46a1-9a6a-d0bf10008503" containerName="payload-processing"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.840998    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841004    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841013    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841018    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841027    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841034    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841040    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841045    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841054    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38380747-222c-4dbe-a1d4-5bee2d1761f8" containerName="payload-processing"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841059    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38380747-222c-4dbe-a1d4-5bee2d1761f8" containerName="payload-processing"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841065    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" containerName="main"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841070    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" containerName="main"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841087    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38" containerName="payload-pre-processing"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841092    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38" containerName="payload-pre-processing"
Sep 17 02:01:53.841139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841147    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="38380747-222c-4dbe-a1d4-5bee2d1761f8" containerName="payload-processing"
Sep 17 02:01:53.841747 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841156    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe0a9a5-0f94-4f6b-8f66-bb84bda3ef38" containerName="payload-pre-processing"
Sep 17 02:01:53.841747 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841164    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy"
Sep 17 02:01:53.841747 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841169    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9bcf9e4-eb3d-4dee-9992-16590bd5d9e0" containerName="main"
Sep 17 02:01:53.841747 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841176    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy"
Sep 17 02:01:53.841747 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841181    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d1ba50-997d-48d3-9208-7d7078e24bd9" containerName="istio-proxy"
Sep 17 02:01:53.841747 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841186    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy"
Sep 17 02:01:53.841747 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841192    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="846b0ac0-49cf-46a1-9a6a-d0bf10008503" containerName="payload-processing"
Sep 17 02:01:53.841747 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.841198    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy"
Sep 17 02:01:53.844007 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.843989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:53.846096 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.846078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4517c39a-kserve-self-signed-certs\""
Sep 17 02:01:53.851066 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.851046    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"]
Sep 17 02:01:53.924263 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.924233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tls-certs\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:53.924263 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.924265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-model-cache\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:53.924450 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.924326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tmp-dir\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:53.924450 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.924349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpg2s\" (UniqueName: \"kubernetes.io/projected/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-kube-api-access-mpg2s\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:53.924450 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.924379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-home\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:53.924450 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:53.924416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-dshm\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.025428 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.025398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-dshm\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.025623 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.025475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tls-certs\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.025623 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.025500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-model-cache\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.025623 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.025589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tmp-dir\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.025805 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.025628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpg2s\" (UniqueName: \"kubernetes.io/projected/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-kube-api-access-mpg2s\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.025805 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.025663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-home\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.025984 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.025956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-model-cache\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.026118 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.026098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tmp-dir\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.026216 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.026194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-home\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.027799 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.027778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-dshm\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.027970 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.027953    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tls-certs\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.033007 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.032987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpg2s\" (UniqueName: \"kubernetes.io/projected/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-kube-api-access-mpg2s\") pod \"e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") " pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.153553 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.153467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:01:54.424013 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:54.423990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"]
Sep 17 02:01:54.425756 ip-10-0-130-87 kubenswrapper[2578]: W0917 02:01:54.425726    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf50dadad_d0a7_44c4_a2c3_9c29abcbb298.slice/crio-9a9888d77372e39ecce44af3cffb32552f45bd579924f069d3f581bc921dbc11 WatchSource:0}: Error finding container 9a9888d77372e39ecce44af3cffb32552f45bd579924f069d3f581bc921dbc11: Status 404 returned error can't find the container with id 9a9888d77372e39ecce44af3cffb32552f45bd579924f069d3f581bc921dbc11
Sep 17 02:01:55.011226 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:55.011188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x" event={"ID":"f50dadad-d0a7-44c4-a2c3-9c29abcbb298","Type":"ContainerStarted","Data":"43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998"}
Sep 17 02:01:55.011226 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:55.011232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x" event={"ID":"f50dadad-d0a7-44c4-a2c3-9c29abcbb298","Type":"ContainerStarted","Data":"9a9888d77372e39ecce44af3cffb32552f45bd579924f069d3f581bc921dbc11"}
Sep 17 02:01:55.028021 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:01:55.027972    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x" podStartSLOduration=2.027957227 podStartE2EDuration="2.027957227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:55.025941948 +0000 UTC m=+2393.696458233" watchObservedRunningTime="2026-09-17 02:01:55.027957227 +0000 UTC m=+2393.698473316"
Sep 17 02:02:04.153842 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:04.153807    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:02:04.154212 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:04.153854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:02:04.166293 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:04.166268    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:02:05.067737 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:05.067694    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:02:35.721544 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:35.721448    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"]
Sep 17 02:02:50.741886 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:50.741823    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x" podUID="f50dadad-d0a7-44c4-a2c3-9c29abcbb298" containerName="main" containerID="cri-o://43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998" gracePeriod=45
Sep 17 02:02:50.991434 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:50.991410    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:02:51.068641 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.068613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tls-certs\") pod \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") "
Sep 17 02:02:51.068825 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.068652    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-home\") pod \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") "
Sep 17 02:02:51.068825 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.068672    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-dshm\") pod \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") "
Sep 17 02:02:51.068825 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.068695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tmp-dir\") pod \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") "
Sep 17 02:02:51.068825 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.068731    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-model-cache\") pod \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") "
Sep 17 02:02:51.068825 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.068776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpg2s\" (UniqueName: \"kubernetes.io/projected/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-kube-api-access-mpg2s\") pod \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\" (UID: \"f50dadad-d0a7-44c4-a2c3-9c29abcbb298\") "
Sep 17 02:02:51.069098 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.068940    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-home" (OuterVolumeSpecName: "home") pod "f50dadad-d0a7-44c4-a2c3-9c29abcbb298" (UID: "f50dadad-d0a7-44c4-a2c3-9c29abcbb298"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 02:02:51.069098 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.068953    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f50dadad-d0a7-44c4-a2c3-9c29abcbb298" (UID: "f50dadad-d0a7-44c4-a2c3-9c29abcbb298"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 02:02:51.069098 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.069014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-model-cache" (OuterVolumeSpecName: "model-cache") pod "f50dadad-d0a7-44c4-a2c3-9c29abcbb298" (UID: "f50dadad-d0a7-44c4-a2c3-9c29abcbb298"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 02:02:51.069250 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.069148    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-home\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 02:02:51.069250 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.069167    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tmp-dir\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 02:02:51.069250 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.069183    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-model-cache\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 02:02:51.070942 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.070907    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-dshm" (OuterVolumeSpecName: "dshm") pod "f50dadad-d0a7-44c4-a2c3-9c29abcbb298" (UID: "f50dadad-d0a7-44c4-a2c3-9c29abcbb298"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 02:02:51.071066 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.070954    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-kube-api-access-mpg2s" (OuterVolumeSpecName: "kube-api-access-mpg2s") pod "f50dadad-d0a7-44c4-a2c3-9c29abcbb298" (UID: "f50dadad-d0a7-44c4-a2c3-9c29abcbb298"). InnerVolumeSpecName "kube-api-access-mpg2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 02:02:51.071066 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.070990    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f50dadad-d0a7-44c4-a2c3-9c29abcbb298" (UID: "f50dadad-d0a7-44c4-a2c3-9c29abcbb298"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 02:02:51.170510 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.170470    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpg2s\" (UniqueName: \"kubernetes.io/projected/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-kube-api-access-mpg2s\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 02:02:51.170510 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.170501    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-tls-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 02:02:51.170510 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.170512    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f50dadad-d0a7-44c4-a2c3-9c29abcbb298-dshm\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 02:02:51.273757 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.273724    2578 generic.go:356] "Generic (PLEG): container finished" podID="f50dadad-d0a7-44c4-a2c3-9c29abcbb298" containerID="43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998" exitCode=0
Sep 17 02:02:51.273935 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.273765    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"
Sep 17 02:02:51.273935 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.273807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x" event={"ID":"f50dadad-d0a7-44c4-a2c3-9c29abcbb298","Type":"ContainerDied","Data":"43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998"}
Sep 17 02:02:51.273935 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.273847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x" event={"ID":"f50dadad-d0a7-44c4-a2c3-9c29abcbb298","Type":"ContainerDied","Data":"9a9888d77372e39ecce44af3cffb32552f45bd579924f069d3f581bc921dbc11"}
Sep 17 02:02:51.273935 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.273865    2578 scope.go:117] "RemoveContainer" containerID="43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998"
Sep 17 02:02:51.283553 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.283520    2578 scope.go:117] "RemoveContainer" containerID="43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998"
Sep 17 02:02:51.283858 ip-10-0-130-87 kubenswrapper[2578]: E0917 02:02:51.283840    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998\": container with ID starting with 43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998 not found: ID does not exist" containerID="43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998"
Sep 17 02:02:51.283932 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.283870    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998"} err="failed to get container status \"43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998\": rpc error: code = NotFound desc = could not find container \"43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998\": container with ID starting with 43253e3cdae5d03f757b92e6525fe4f1f65ba035dabffdbfe33ea6fe45528998 not found: ID does not exist"
Sep 17 02:02:51.296651 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.296618    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"]
Sep 17 02:02:51.299986 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.299962    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4517c39a-kserve-67b8f5bf8c-4486x"]
Sep 17 02:02:51.919868 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:02:51.919835    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f50dadad-d0a7-44c4-a2c3-9c29abcbb298" path="/var/lib/kubelet/pods/f50dadad-d0a7-44c4-a2c3-9c29abcbb298/volumes"
Sep 17 02:03:03.934940 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.934906    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"]
Sep 17 02:03:03.935313 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.935266    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy"
Sep 17 02:03:03.935313 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.935278    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7901ff35-db82-4476-8a59-acd18a605682" containerName="istio-proxy"
Sep 17 02:03:03.935313 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.935306    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f50dadad-d0a7-44c4-a2c3-9c29abcbb298" containerName="main"
Sep 17 02:03:03.935313 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.935315    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f50dadad-d0a7-44c4-a2c3-9c29abcbb298" containerName="main"
Sep 17 02:03:03.935578 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.935369    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a457cf9-d9c1-489d-b589-9afb8edeb52c" containerName="istio-proxy"
Sep 17 02:03:03.935578 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.935377    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f50dadad-d0a7-44c4-a2c3-9c29abcbb298" containerName="main"
Sep 17 02:03:03.939915 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.939892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:03.949115 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.949087    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"]
Sep 17 02:03:03.998329 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.998297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-metrics-certs\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:03.998515 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.998358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6bl\" (UniqueName: \"kubernetes.io/projected/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-kube-api-access-tc6bl\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:03.998515 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.998406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-openshift-service-ca\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:03.998515 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:03.998455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-maas-api-tls\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:04.099805 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.099770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6bl\" (UniqueName: \"kubernetes.io/projected/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-kube-api-access-tc6bl\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:04.099995 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.099823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-openshift-service-ca\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:04.099995 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.099855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-maas-api-tls\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:04.099995 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.099921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-metrics-certs\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:04.100524 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.100493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-openshift-service-ca\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:04.102434 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.102410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-maas-api-tls\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:04.102563 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.102466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-metrics-certs\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:04.110377 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.110358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6bl\" (UniqueName: \"kubernetes.io/projected/5b85c2bb-cc9b-42b0-8a6a-3194dcef399b-kube-api-access-tc6bl\") pod \"maas-api-86d9ddc875-4v7jj\" (UID: \"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:04.252755 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.252672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:04.524971 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.524943    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"]
Sep 17 02:03:04.526759 ip-10-0-130-87 kubenswrapper[2578]: W0917 02:03:04.526736    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b85c2bb_cc9b_42b0_8a6a_3194dcef399b.slice/crio-70c1421d6b1cbe8c15b829affe5ef3366773a1c922340adc211a8b66fd650a9e WatchSource:0}: Error finding container 70c1421d6b1cbe8c15b829affe5ef3366773a1c922340adc211a8b66fd650a9e: Status 404 returned error can't find the container with id 70c1421d6b1cbe8c15b829affe5ef3366773a1c922340adc211a8b66fd650a9e
Sep 17 02:03:04.528076 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:04.528059    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 02:03:05.340919 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:05.340881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj" event={"ID":"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b","Type":"ContainerStarted","Data":"c455fb0041c91ce2f0fcc80e1d550fe25dcf0e86c29f72dbafd7982c2cea8c4e"}
Sep 17 02:03:05.341267 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:05.340943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj" event={"ID":"5b85c2bb-cc9b-42b0-8a6a-3194dcef399b","Type":"ContainerStarted","Data":"70c1421d6b1cbe8c15b829affe5ef3366773a1c922340adc211a8b66fd650a9e"}
Sep 17 02:03:05.341267 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:05.340971    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:05.357347 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:05.357298    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj" podStartSLOduration=2.054714425 podStartE2EDuration="2.357280983s" totalImagesPullingTime="302.566558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:03:05.355610057 +0000 UTC m=+2464.026126147" watchObservedRunningTime="2026-09-17 02:03:05.357280983 +0000 UTC m=+2464.027797072"
Sep 17 02:03:11.351587 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:11.351559    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4v7jj"
Sep 17 02:03:11.397196 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:11.397164    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"]
Sep 17 02:03:11.397466 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:11.397419    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r" podUID="0590fb3d-04f3-4064-a911-3fc95c92d7d1" containerName="maas-api" containerID="cri-o://79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628" gracePeriod=30
Sep 17 02:03:12.640742 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.640718    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 02:03:12.780675 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.780591    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0590fb3d-04f3-4064-a911-3fc95c92d7d1-openshift-service-ca\") pod \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") "
Sep 17 02:03:12.780675 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.780648    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-metrics-certs\") pod \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") "
Sep 17 02:03:12.780916 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.780709    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls5gt\" (UniqueName: \"kubernetes.io/projected/0590fb3d-04f3-4064-a911-3fc95c92d7d1-kube-api-access-ls5gt\") pod \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") "
Sep 17 02:03:12.780916 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.780769    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-maas-api-tls\") pod \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\" (UID: \"0590fb3d-04f3-4064-a911-3fc95c92d7d1\") "
Sep 17 02:03:12.781032 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.781002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0590fb3d-04f3-4064-a911-3fc95c92d7d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0590fb3d-04f3-4064-a911-3fc95c92d7d1" (UID: "0590fb3d-04f3-4064-a911-3fc95c92d7d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 02:03:12.781139 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.781119    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0590fb3d-04f3-4064-a911-3fc95c92d7d1-openshift-service-ca\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 02:03:12.782926 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.782892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0590fb3d-04f3-4064-a911-3fc95c92d7d1" (UID: "0590fb3d-04f3-4064-a911-3fc95c92d7d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 02:03:12.783038 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.782948    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0590fb3d-04f3-4064-a911-3fc95c92d7d1-kube-api-access-ls5gt" (OuterVolumeSpecName: "kube-api-access-ls5gt") pod "0590fb3d-04f3-4064-a911-3fc95c92d7d1" (UID: "0590fb3d-04f3-4064-a911-3fc95c92d7d1"). InnerVolumeSpecName "kube-api-access-ls5gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 02:03:12.783038 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.782988    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0590fb3d-04f3-4064-a911-3fc95c92d7d1" (UID: "0590fb3d-04f3-4064-a911-3fc95c92d7d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 02:03:12.881621 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.881589    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-metrics-certs\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 02:03:12.881621 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.881618    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls5gt\" (UniqueName: \"kubernetes.io/projected/0590fb3d-04f3-4064-a911-3fc95c92d7d1-kube-api-access-ls5gt\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 02:03:12.881621 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:12.881629    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0590fb3d-04f3-4064-a911-3fc95c92d7d1-maas-api-tls\") on node \"ip-10-0-130-87.ec2.internal\" DevicePath \"\""
Sep 17 02:03:13.382747 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:13.382715    2578 generic.go:356] "Generic (PLEG): container finished" podID="0590fb3d-04f3-4064-a911-3fc95c92d7d1" containerID="79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628" exitCode=0
Sep 17 02:03:13.382920 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:13.382777    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"
Sep 17 02:03:13.382920 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:13.382796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r" event={"ID":"0590fb3d-04f3-4064-a911-3fc95c92d7d1","Type":"ContainerDied","Data":"79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628"}
Sep 17 02:03:13.382920 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:13.382849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r" event={"ID":"0590fb3d-04f3-4064-a911-3fc95c92d7d1","Type":"ContainerDied","Data":"175e01053bd1e338fa597f4b61164d49e679f6f44e0827f3444b1f26c2b151e2"}
Sep 17 02:03:13.382920 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:13.382873    2578 scope.go:117] "RemoveContainer" containerID="79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628"
Sep 17 02:03:13.392839 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:13.392820    2578 scope.go:117] "RemoveContainer" containerID="79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628"
Sep 17 02:03:13.393123 ip-10-0-130-87 kubenswrapper[2578]: E0917 02:03:13.393104    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628\": container with ID starting with 79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628 not found: ID does not exist" containerID="79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628"
Sep 17 02:03:13.393164 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:13.393133    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628"} err="failed to get container status \"79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628\": rpc error: code = NotFound desc = could not find container \"79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628\": container with ID starting with 79b7f662478a78173471ecce81de894d51b4ea656f2b97b3afc7c5ae77e7c628 not found: ID does not exist"
Sep 17 02:03:13.403359 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:13.403333    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"]
Sep 17 02:03:13.406882 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:13.406861    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-gs27r"]
Sep 17 02:03:13.920635 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:13.920607    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0590fb3d-04f3-4064-a911-3fc95c92d7d1" path="/var/lib/kubelet/pods/0590fb3d-04f3-4064-a911-3fc95c92d7d1/volumes"
Sep 17 02:03:29.224313 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:29.224281    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mws4d_a9850b1d-04e3-4bbd-9072-a9ac2dab518a/global-pull-secret-syncer/0.log"
Sep 17 02:03:29.335645 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:29.335616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r9hm4_1a2a1999-931a-46a1-83b0-57b1c9ca0dda/konnectivity-agent/0.log"
Sep 17 02:03:29.353578 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:29.353552    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-87.ec2.internal_e047cec48a9cc0053170912f91d4d94a/haproxy/0.log"
Sep 17 02:03:34.633240 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:34.633207    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69bcd7c777-xf8br_b0008dcc-a37c-4709-8c6e-e7eeb6916b9a/authorino/0.log"
Sep 17 02:03:34.665224 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:34.665195    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-8bfsk_69cd6c13-d387-474e-80d2-0649d13cfc49/manager/0.log"
Sep 17 02:03:36.583764 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:36.583731    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-ffmtx_a43fe2d1-5671-489e-803e-6f3920eb9cea/cluster-monitoring-operator/0.log"
Sep 17 02:03:36.607617 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:36.607584    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bbp4g_e28578e8-6ade-432c-89e7-fb253a1d8066/kube-state-metrics/0.log"
Sep 17 02:03:36.623204 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:36.623182    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bbp4g_e28578e8-6ade-432c-89e7-fb253a1d8066/kube-rbac-proxy-main/0.log"
Sep 17 02:03:36.655833 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:36.655811    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-bbp4g_e28578e8-6ade-432c-89e7-fb253a1d8066/kube-rbac-proxy-self/0.log"
Sep 17 02:03:36.681284 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:36.681256    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84754b4c75-25xht_912ec005-99b4-448a-b189-53ddd78f1893/metrics-server/0.log"
Sep 17 02:03:36.710328 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:36.710305    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-rzgsm_57bc09b2-56af-49c2-9e8f-52c190a2a917/monitoring-plugin/0.log"
Sep 17 02:03:36.873295 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:36.873223    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq7pq_bb544b95-5cf3-4468-9fd7-58f785c29c60/node-exporter/0.log"
Sep 17 02:03:36.894497 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:36.894477    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq7pq_bb544b95-5cf3-4468-9fd7-58f785c29c60/kube-rbac-proxy/0.log"
Sep 17 02:03:36.919254 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:36.919226    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq7pq_bb544b95-5cf3-4468-9fd7-58f785c29c60/init-textfile/0.log"
Sep 17 02:03:37.102173 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:37.102148    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6072a131-09b6-4e4a-9508-b240e512f563/prometheus/0.log"
Sep 17 02:03:37.131555 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:37.131453    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6072a131-09b6-4e4a-9508-b240e512f563/config-reloader/0.log"
Sep 17 02:03:37.154101 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:37.154073    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6072a131-09b6-4e4a-9508-b240e512f563/thanos-sidecar/0.log"
Sep 17 02:03:37.188838 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:37.188814    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6072a131-09b6-4e4a-9508-b240e512f563/kube-rbac-proxy-web/0.log"
Sep 17 02:03:37.212097 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:37.212071    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6072a131-09b6-4e4a-9508-b240e512f563/kube-rbac-proxy/0.log"
Sep 17 02:03:37.240666 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:37.240643    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6072a131-09b6-4e4a-9508-b240e512f563/kube-rbac-proxy-thanos/0.log"
Sep 17 02:03:37.263823 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:37.263802    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6072a131-09b6-4e4a-9508-b240e512f563/init-config-reloader/0.log"
Sep 17 02:03:37.311597 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:37.311555    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-4j6jh_1b3cb156-e78f-45e4-af20-6684d1875254/prometheus-operator/0.log"
Sep 17 02:03:37.366908 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:37.366881    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-4j6jh_1b3cb156-e78f-45e4-af20-6684d1875254/kube-rbac-proxy/0.log"
Sep 17 02:03:39.535143 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.535108    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"]
Sep 17 02:03:39.535620 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.535506    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0590fb3d-04f3-4064-a911-3fc95c92d7d1" containerName="maas-api"
Sep 17 02:03:39.535620 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.535524    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0590fb3d-04f3-4064-a911-3fc95c92d7d1" containerName="maas-api"
Sep 17 02:03:39.535725 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.535650    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0590fb3d-04f3-4064-a911-3fc95c92d7d1" containerName="maas-api"
Sep 17 02:03:39.538518 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.538501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.547064 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.541284    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r7tqq\"/\"kube-root-ca.crt\""
Sep 17 02:03:39.547064 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.541690    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r7tqq\"/\"openshift-service-ca.crt\""
Sep 17 02:03:39.547064 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.541955    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r7tqq\"/\"default-dockercfg-t8545\""
Sep 17 02:03:39.547361 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.547335    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"]
Sep 17 02:03:39.626397 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.626366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-sys\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.626592 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.626408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-podres\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.626592 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.626489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-proc\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.626592 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.626522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-lib-modules\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.626592 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.626573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rzb\" (UniqueName: \"kubernetes.io/projected/074e4315-aaad-4243-9bc7-6a0ca2a13b34-kube-api-access-w6rzb\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.727086 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.727046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-lib-modules\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.727086 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.727090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rzb\" (UniqueName: \"kubernetes.io/projected/074e4315-aaad-4243-9bc7-6a0ca2a13b34-kube-api-access-w6rzb\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.727322 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.727142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-sys\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.727322 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.727170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-podres\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.727322 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.727224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-proc\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.727322 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.727239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-lib-modules\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.727322 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.727279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-sys\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.727322 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.727295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-proc\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.727521 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.727326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/074e4315-aaad-4243-9bc7-6a0ca2a13b34-podres\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.734724 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.734697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rzb\" (UniqueName: \"kubernetes.io/projected/074e4315-aaad-4243-9bc7-6a0ca2a13b34-kube-api-access-w6rzb\") pod \"perf-node-gather-daemonset-ftzg2\" (UID: \"074e4315-aaad-4243-9bc7-6a0ca2a13b34\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:39.855457 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:39.855426    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:40.121708 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:40.121624    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"]
Sep 17 02:03:40.124685 ip-10-0-130-87 kubenswrapper[2578]: W0917 02:03:40.124653    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod074e4315_aaad_4243_9bc7_6a0ca2a13b34.slice/crio-70c2f1c6f3c68dc7935c2af8ea7ef8319baefe6ce31063b8ecb0b5e1aae817fe WatchSource:0}: Error finding container 70c2f1c6f3c68dc7935c2af8ea7ef8319baefe6ce31063b8ecb0b5e1aae817fe: Status 404 returned error can't find the container with id 70c2f1c6f3c68dc7935c2af8ea7ef8319baefe6ce31063b8ecb0b5e1aae817fe
Sep 17 02:03:40.515330 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:40.515238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2" event={"ID":"074e4315-aaad-4243-9bc7-6a0ca2a13b34","Type":"ContainerStarted","Data":"ca19eeced4545e39827016be8ca62cde154b519c13b1869b66d308c8590e3a82"}
Sep 17 02:03:40.515330 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:40.515293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2" event={"ID":"074e4315-aaad-4243-9bc7-6a0ca2a13b34","Type":"ContainerStarted","Data":"70c2f1c6f3c68dc7935c2af8ea7ef8319baefe6ce31063b8ecb0b5e1aae817fe"}
Sep 17 02:03:40.515330 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:40.515315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:40.530592 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:40.530520    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2" podStartSLOduration=1.530506342 podStartE2EDuration="1.530506342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 02:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:03:40.529622917 +0000 UTC m=+2499.200139006" watchObservedRunningTime="2026-09-17 02:03:40.530506342 +0000 UTC m=+2499.201022431"
Sep 17 02:03:41.959483 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:41.959458    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nrzkz_123f13ef-59c4-4635-8550-ee114b9d1b7b/dns/0.log"
Sep 17 02:03:41.977862 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:41.977841    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nrzkz_123f13ef-59c4-4635-8550-ee114b9d1b7b/kube-rbac-proxy/0.log"
Sep 17 02:03:42.023676 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:42.023642    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mxzkv_d2d5b623-d9d2-4621-bc65-66241a002022/dns-node-resolver/0.log"
Sep 17 02:03:42.535317 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:42.535286    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f8798949b-9v5sv_975e24fc-f293-4864-98b3-caef33a0e4ec/registry/0.log"
Sep 17 02:03:42.553971 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:42.553949    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-22cxn_33989fb0-89ab-47a8-90a4-2fbd2332d884/node-ca/0.log"
Sep 17 02:03:43.879640 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:43.879608    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c5f54bcb9-dtvxf_b6b58fa3-6612-4601-b02c-c35e7e95bad2/kube-auth-proxy/0.log"
Sep 17 02:03:44.866989 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:44.866949    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pdvgq_c7e25797-9a29-4a55-b0d9-bb5b89699bae/serve-healthcheck-canary/0.log"
Sep 17 02:03:45.310835 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:45.310800    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-qhwv5_eae6b120-681a-4868-b818-151648b29629/insights-operator/0.log"
Sep 17 02:03:45.311294 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:45.311005    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-qhwv5_eae6b120-681a-4868-b818-151648b29629/insights-operator/1.log"
Sep 17 02:03:45.469079 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:45.469053    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-twqpd_4fb7434b-5484-4868-a9e2-78f64165b632/kube-rbac-proxy/0.log"
Sep 17 02:03:45.486951 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:45.486931    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-twqpd_4fb7434b-5484-4868-a9e2-78f64165b632/exporter/0.log"
Sep 17 02:03:45.507392 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:45.507367    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-twqpd_4fb7434b-5484-4868-a9e2-78f64165b632/extractor/0.log"
Sep 17 02:03:46.528898 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:46.528867    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-ftzg2"
Sep 17 02:03:48.379213 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:48.379176    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57c6978585-z7z4j_b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6/manager/0.log"
Sep 17 02:03:48.402201 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:48.402175    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57c6978585-z7z4j_b24cfb81-d23a-4bc0-aa2a-fab4d97e73a6/copy-manifests/0.log"
Sep 17 02:03:49.887326 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:49.887300    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5749dbf97f-plnxb_166dd18d-8407-4405-8a6f-7e9085d75e04/manager/0.log"
Sep 17 02:03:50.784282 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:50.784206    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c5f54bcb9-dtvxf_b6b58fa3-6612-4601-b02c-c35e7e95bad2/kube-auth-proxy/0.log"
Sep 17 02:03:52.701135 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:52.701106    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c5f54bcb9-dtvxf_b6b58fa3-6612-4601-b02c-c35e7e95bad2/kube-auth-proxy/0.log"
Sep 17 02:03:56.352740 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:56.352715    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvtgt_7c5faf65-3a14-47b8-8077-af57e33f8dfc/kube-multus-additional-cni-plugins/0.log"
Sep 17 02:03:56.370326 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:56.370300    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvtgt_7c5faf65-3a14-47b8-8077-af57e33f8dfc/egress-router-binary-copy/0.log"
Sep 17 02:03:56.388436 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:56.388410    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvtgt_7c5faf65-3a14-47b8-8077-af57e33f8dfc/cni-plugins/0.log"
Sep 17 02:03:56.406813 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:56.406792    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvtgt_7c5faf65-3a14-47b8-8077-af57e33f8dfc/bond-cni-plugin/0.log"
Sep 17 02:03:56.425364 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:56.425340    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvtgt_7c5faf65-3a14-47b8-8077-af57e33f8dfc/routeoverride-cni/0.log"
Sep 17 02:03:56.443744 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:56.443721    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvtgt_7c5faf65-3a14-47b8-8077-af57e33f8dfc/whereabouts-cni-bincopy/0.log"
Sep 17 02:03:56.462328 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:56.462309    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvtgt_7c5faf65-3a14-47b8-8077-af57e33f8dfc/whereabouts-cni/0.log"
Sep 17 02:03:56.845251 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:56.845227    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zdnlz_e4ba1fbb-6994-4cd1-a0db-2ac0299ef57d/kube-multus/0.log"
Sep 17 02:03:56.906050 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:56.906024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-blh4d_ddb0ced7-abc3-439b-b250-ff90f01679cc/network-metrics-daemon/0.log"
Sep 17 02:03:56.921354 ip-10-0-130-87 kubenswrapper[2578]: I0917 02:03:56.921333    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-blh4d_ddb0ced7-abc3-439b-b250-ff90f01679cc/kube-rbac-proxy/0.log"