Sep 06 10:21:50.144011 ip-10-0-130-112 systemd[1]: Starting Kubernetes Kubelet...
Sep 06 10:21:50.531705 ip-10-0-130-112 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 10:21:50.531705 ip-10-0-130-112 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 06 10:21:50.531705 ip-10-0-130-112 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 10:21:50.531705 ip-10-0-130-112 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 06 10:21:50.531705 ip-10-0-130-112 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 10:21:50.533167 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.533078    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 06 10:21:50.537983 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.537962    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:50.537983 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.537980    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:50.537983 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.537985    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:50.537983 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.537989    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.537993    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538001    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538004    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538007    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538010    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538012    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538015    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538018    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538021    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538024    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538027    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538029    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538032    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538035    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538046    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538049    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538052    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538055    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538058    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:50.538150 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538061    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538064    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538067    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538070    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538073    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538075    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538078    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538085    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538089    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538092    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538095    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538097    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538100    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538105    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538109    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538112    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538115    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538117    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538120    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:50.538655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538125    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538128    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538131    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538134    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538137    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538140    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538143    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538146    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538148    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538151    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538154    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538157    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538161    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538171    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538174    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538177    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538179    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538182    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538185    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:50.539133 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538188    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538191    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538194    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538197    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538199    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538202    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538205    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538210    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538215    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538219    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538223    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538226    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538229    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538232    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538235    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538237    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538240    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538243    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538250    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538252    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:50.539621 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538255    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538258    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538260    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538263    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538266    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538269    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538272    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538274    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538277    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538280    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538285    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538288    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538290    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538293    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538296    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538299    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538303    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.538307    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539492    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539500    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:50.540115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539504    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539513    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539516    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539520    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539523    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539526    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539529    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539532    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539535    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539541    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539544    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539547    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539550    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539553    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539555    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539558    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539561    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539564    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539567    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:50.540619 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539570    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539572    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539575    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539580    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539583    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539586    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539588    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539591    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539593    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539596    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539599    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539602    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539605    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539608    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539611    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539621    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539625    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539627    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539630    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539633    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:50.541115 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539636    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539639    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539643    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539646    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539648    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539651    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539654    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539658    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539661    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539664    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539667    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539669    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539672    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539675    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539677    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539680    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539683    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539686    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539688    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539691    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:50.541655 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539696    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539699    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539702    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539705    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539726    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539730    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539733    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539737    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539748    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539752    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539755    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539758    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539763    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539767    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539770    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539772    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539777    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539780    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539782    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:50.542182 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539785    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539789    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539793    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539797    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539803    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539806    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539809    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539813    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539815    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539818    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539821    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539824    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539827    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539829    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539832    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539835    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539838    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539843    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.539846    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:50.542658 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540542    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540549    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540559    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540562    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540565    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540569    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540575    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540578    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540581    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540584    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540587    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540590    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540593    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540595    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540598    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540600    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540603    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540606    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540609    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:50.543136 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540614    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540617    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540620    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540623    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540626    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540628    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540631    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540635    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540638    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540642    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540644    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540647    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540650    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540655    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540658    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540660    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540668    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540671    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540674    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540676    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:50.543593 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540679    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540682    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540684    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540687    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540690    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540695    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540697    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540700    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540703    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540724    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540728    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540730    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540733    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540736    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540739    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540744    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540748    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540754    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540757    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:50.544099 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540760    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540763    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540766    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540769    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540772    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540775    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540778    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540781    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540783    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540786    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540798    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540801    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540803    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540806    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540809    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540812    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540814    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540817    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540820    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540823    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:50.544562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540825    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540828    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540831    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540837    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540840    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540843    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540845    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540848    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540851    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540854    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540856    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540859    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540862    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540864    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540867    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540873    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540875    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540878    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540881    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540883    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.540886    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:50.545078 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541216    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541259    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541277    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541283    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541288    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541293    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541299    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541304    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541307    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541312    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541315    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541321    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541326    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541330    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541334    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541337    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541340    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541343    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541348    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541351    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541354    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541357    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541361    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541365    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 06 10:21:50.545606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541369    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541372    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541376    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541379    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541382    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541384    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541388    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541391    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541404    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541408    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541411    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541414    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541417    2564 flags.go:64] FLAG: --enable-server="true"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541420    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541425    2564 flags.go:64] FLAG: --event-burst="100"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541428    2564 flags.go:64] FLAG: --event-qps="50"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541431    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541434    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541437    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541441    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541444    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541447    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541450    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541453    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541456    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 06 10:21:50.546292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541460    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541463    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541465    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541468    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541471    2564 flags.go:64] FLAG: --feature-gates=""
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541475    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541478    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541481    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541484    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541487    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541490    2564 flags.go:64] FLAG: --help="false"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541493    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-130-112.ec2.internal"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541504    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541508    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541511    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541516    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541520    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541523    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541526    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541529    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541532    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541535    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541538    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541541    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 06 10:21:50.546976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541544    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541547    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541550    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541553    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541556    2564 flags.go:64] FLAG: --lock-file=""
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541559    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541562    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541565    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541570    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541574    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541577    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541580    2564 flags.go:64] FLAG: --logging-format="text"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541583    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541586    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541589    2564 flags.go:64] FLAG: --manifest-url=""
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541592    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541596    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541599    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541604    2564 flags.go:64] FLAG: --max-pods="110"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541607    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541610    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541613    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541616    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541620    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541623    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 06 10:21:50.547568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541626    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541635    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541638    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541641    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541644    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541647    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541652    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541655    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541658    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541661    2564 flags.go:64] FLAG: --port="10250"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541664    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541668    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0457dc59dfb757577"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541671    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541674    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541677    2564 flags.go:64] FLAG: --register-node="true"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541680    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541684    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541687    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541690    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541693    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541697    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541700    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541703    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541706    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541720    2564 flags.go:64] FLAG: --runonce="false"
Sep 06 10:21:50.548183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541723    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541726    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541731    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541734    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541737    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541741    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541745    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541748    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541751    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541754    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541757    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541759    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541763    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541766    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541769    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541775    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541778    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541781    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541785    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541788    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541791    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541794    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541797    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541800    2564 flags.go:64] FLAG: --v="2"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541804    2564 flags.go:64] FLAG: --version="false"
Sep 06 10:21:50.548821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541809    2564 flags.go:64] FLAG: --vmodule=""
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541812    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.541816    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541957    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541961    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541964    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541967    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541971    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541974    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541980    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541985    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541989    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541993    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541996    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.541998    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542001    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542004    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542007    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542009    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:50.549438 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542012    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542015    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542017    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542020    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542023    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542026    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542028    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542032    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542035    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542038    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542040    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542043    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542045    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542048    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542051    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542053    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542056    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542059    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542061    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:50.550007 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542064    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542066    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542069    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542073    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542075    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542078    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542082    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542085    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542087    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542090    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542093    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542095    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542098    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542100    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542103    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542105    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542108    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542110    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542113    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542116    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:50.550512 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542118    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542121    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542124    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542126    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542129    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542131    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542134    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542136    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542139    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542142    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542145    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542147    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542150    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542152    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542155    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542158    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542161    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542164    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542168    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542171    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:50.551017 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542173    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542176    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542178    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542181    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542184    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542187    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542190    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542192    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542195    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542198    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542200    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542203    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542205    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542208    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542211    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542214    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542216    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542220    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542224    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542227    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:50.551503 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542229    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542232    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542235    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.542238    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.542844    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.550091    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.550107    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.550126    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.550132    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550195    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550201    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550204    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550207    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550211    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:50.552009 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550215    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550219    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550222    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550225    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550228    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550231    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550234    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550237    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550240    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550243    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550246    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550250    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550254    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550257    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550260    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550262    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550265    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550268    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550270    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:50.552357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550274    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550276    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550279    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550281    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550284    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550287    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550290    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550293    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550296    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550299    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550301    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550304    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550306    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550309    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550311    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550314    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550316    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550319    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550322    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550324    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:50.552843 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550327    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550330    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550332    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550335    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550338    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550341    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550344    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550347    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550349    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550352    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550354    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550357    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550360    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550364    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550366    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550369    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550372    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550375    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550377    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550380    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:50.553331 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550382    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550385    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550388    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550390    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550393    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550395    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550398    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550401    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550403    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550406    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550408    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550412    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550416    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550418    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550422    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550424    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550427    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550430    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550433    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550435    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:50.553887 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550438    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550440    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550443    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550445    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550448    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550451    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550454    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550457    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550459    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550462    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550464    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550467    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550470    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550473    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:50.554399 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550475    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.550480    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550614    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550618    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550622    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550626    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550629    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550632    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550635    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550638    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550641    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550644    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550647    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550649    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550652    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:50.554760 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550655    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550657    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550660    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550662    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550665    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550667    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550670    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550672    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550675    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550678    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550681    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550684    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550686    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550689    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550692    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550695    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550697    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550700    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550702    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550720    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:50.555129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550725    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550729    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550732    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550734    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550737    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550740    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550742    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550745    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550748    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550751    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550754    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550756    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550759    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550761    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550764    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550767    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550769    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550772    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550775    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:50.555615 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550778    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550780    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550783    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550786    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550792    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550795    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550798    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550800    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550803    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550805    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550808    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550810    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550813    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550816    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550818    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550821    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550823    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550827    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550831    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:50.556097 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550833    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550836    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550839    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550842    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550845    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550847    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550850    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550853    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550855    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550858    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550860    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550863    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550866    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550868    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550871    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550873    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550876    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550879    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550884    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550887    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:50.556556 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550889    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550892    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550894    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550897    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550899    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550902    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550904    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:50.550907    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.550912    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.551016    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.553833    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.554914    2564 server.go:1019] "Starting client certificate rotation"
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.555019    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 10:21:50.557054 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.555052    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 10:21:50.559299 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.559285    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 10:21:50.560634 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.560611    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 10:21:50.572462 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.572441    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 06 10:21:50.578573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.578141    2564 log.go:25] "Validated CRI v1 image API"
Sep 06 10:21:50.580019 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.580001    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 06 10:21:50.582488 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.582468    2564 fs.go:135] Filesystem UUIDs: map[44b7848e-88a4-4cf8-88bb-bd432f9157a2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c4b15ff2-7f36-4b30-9225-ae1e36887985:/dev/nvme0n1p3]
Sep 06 10:21:50.582548 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.582489    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 06 10:21:50.588304 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.588167    2564 manager.go:216] Machine: {Timestamp:2026-09-06 10:21:50.587028441 +0000 UTC m=+0.336030185 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098322 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2552563f31d79663c54a97f02b8587 SystemUUID:ec255256-3f31-d796-63c5-4a97f02b8587 BootID:66f8edcb-ff67-4bdf-ba77-6d8abc20f3a9 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:03:0e:d5:2d:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:03:0e:d5:2d:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:d6:af:1f:a0:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 06 10:21:50.588304 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.588290    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 06 10:21:50.588449 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.588369    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 06 10:21:50.589625 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.589609    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 10:21:50.591410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.591384    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 06 10:21:50.591568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.591414    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-112.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 06 10:21:50.592090 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.592080    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 06 10:21:50.592127 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.592093    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 06 10:21:50.592127 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.592101    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 10:21:50.592781 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.592771    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 10:21:50.594129 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.594119    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 06 10:21:50.596887 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.596876    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 10:21:50.597031 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.597022    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 06 10:21:50.597059 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.597051    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 06 10:21:50.597092 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.597062    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 06 10:21:50.597092 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.597080    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 06 10:21:50.597092 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.597090    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 06 10:21:50.597173 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.597099    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 06 10:21:50.597440 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.597429    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 10:21:50.597481 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.597447    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 10:21:50.599689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.599662    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 06 10:21:50.601859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.601844    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 06 10:21:50.601930 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.601877    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 06 10:21:50.602098 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602088    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 06 10:21:50.602128 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602103    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 06 10:21:50.602128 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602109    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 06 10:21:50.602128 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602114    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 06 10:21:50.602128 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602124    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 06 10:21:50.602269 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602132    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 06 10:21:50.602867 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602858    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 06 10:21:50.602867 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602867    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 06 10:21:50.602943 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602876    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 06 10:21:50.602943 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602884    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 06 10:21:50.602943 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602895    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 06 10:21:50.602943 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.602902    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 06 10:21:50.604420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.604406    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 06 10:21:50.604420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.604417    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 06 10:21:50.605219 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.605190    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 06 10:21:50.605219 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.605188    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-112.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 06 10:21:50.607898 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.607883    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-112.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 06 10:21:50.608261 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.608248    2564 server.go:1268] "Started kubelet"
Sep 06 10:21:50.608357 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.608331    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 06 10:21:50.608443 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.608399    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 06 10:21:50.608483 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.608470    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 06 10:21:50.609253 ip-10-0-130-112 systemd[1]: Started Kubernetes Kubelet.
Sep 06 10:21:50.609553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.609536    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 06 10:21:50.611249 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.611232    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 06 10:21:50.611353 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.611326    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pz25d"
Sep 06 10:21:50.616451 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.615566    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-112.ec2.internal.18d2b42dc00e9d1a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-112.ec2.internal,UID:ip-10-0-130-112.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-112.ec2.internal,},FirstTimestamp:2026-09-06 10:21:50.608219418 +0000 UTC m=+0.357221142,LastTimestamp:2026-09-06 10:21:50.608219418 +0000 UTC m=+0.357221142,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-112.ec2.internal,}"
Sep 06 10:21:50.616664 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.616645    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 06 10:21:50.616837 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.616823    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 06 10:21:50.616890 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.616837    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 06 10:21:50.617033 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.616903    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:50.617833 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.617416    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 06 10:21:50.617833 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.617417    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 06 10:21:50.617833 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.617443    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 06 10:21:50.617833 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.617617    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 06 10:21:50.617833 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.617629    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 06 10:21:50.618297 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.618281    2564 factory.go:55] Registering systemd factory
Sep 06 10:21:50.618361 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.618327    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 06 10:21:50.618541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.618527    2564 factory.go:153] Registering CRI-O factory
Sep 06 10:21:50.618623 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.618543    2564 factory.go:223] Registration of the crio container factory successfully
Sep 06 10:21:50.618623 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.618591    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 06 10:21:50.618623 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.618612    2564 factory.go:103] Registering Raw factory
Sep 06 10:21:50.618623 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.618623    2564 manager.go:1196] Started watching for new ooms in manager
Sep 06 10:21:50.618978 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.618961    2564 manager.go:349] Starting recovery of all containers
Sep 06 10:21:50.619836 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.619815    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pz25d"
Sep 06 10:21:50.620162 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.620138    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-112.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 06 10:21:50.620287 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.620266    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 06 10:21:50.624539 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.624511    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 06 10:21:50.631148 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.631128    2564 manager.go:354] Recovery completed
Sep 06 10:21:50.637598 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.637444    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:50.639384 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.639367    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:50.639445 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.639407    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:50.639445 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.639421    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:50.640105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.640084    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 06 10:21:50.640105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.640096    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 06 10:21:50.640216 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.640117    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 10:21:50.640216 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.640099    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:50.640981 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.640965    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:50.641069 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.640996    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:50.641069 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.641010    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:50.641265 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.641253    2564 policy_none.go:49] "None policy: Start"
Sep 06 10:21:50.641314 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.641269    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 06 10:21:50.641314 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.641279    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 06 10:21:50.642341 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.642331    2564 policy_none.go:47] "Start"
Sep 06 10:21:50.672397 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.672382    2564 manager.go:337] "Starting Device Plugin manager"
Sep 06 10:21:50.690689 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.672421    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 06 10:21:50.690689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.672434    2564 server.go:87] "Starting device plugin registration server"
Sep 06 10:21:50.690689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.672774    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 06 10:21:50.690689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.672785    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 06 10:21:50.690689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.673041    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 06 10:21:50.690689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.673124    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 06 10:21:50.690689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.673134    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 06 10:21:50.690689 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.673874    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 06 10:21:50.690689 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.673926    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:50.756432 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.756406    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 06 10:21:50.756566 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.756463    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 06 10:21:50.756566 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.756488    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 06 10:21:50.756658 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.756595    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 06 10:21:50.759359 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.759339    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:50.773477 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.773454    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:50.774374 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.774356    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:50.774471 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.774392    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:50.774471 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.774406    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:50.774471 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.774436    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-112.ec2.internal"
Sep 06 10:21:50.782695 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.782619    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-112.ec2.internal"
Sep 06 10:21:50.782695 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.782649    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-112.ec2.internal\": node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:50.794258 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.794238    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:50.857554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.857519    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal"]
Sep 06 10:21:50.857670 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.857594    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:50.859122 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.859106    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:50.859195 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.859138    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:50.859195 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.859152    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:50.860605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.860592    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:50.860766 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.860751    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:50.860823 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.860782    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:50.861312 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.861295    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:50.861378 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.861329    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:50.861378 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.861340    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:50.861435 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.861295    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:50.861435 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.861405    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:50.861435 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.861417    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:50.862632 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.862618    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:50.862694 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.862643    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:50.863269 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.863254    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:50.863342 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.863287    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:50.863342 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.863300    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:50.878217 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.878199    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-112.ec2.internal\" not found" node="ip-10-0-130-112.ec2.internal"
Sep 06 10:21:50.881977 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.881963    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-112.ec2.internal\" not found" node="ip-10-0-130-112.ec2.internal"
Sep 06 10:21:50.894933 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.894920    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:50.919785 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.919766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aaa7e60bc565da98eaf30ac97210db8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"aaa7e60bc565da98eaf30ac97210db8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:50.919860 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.919790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaa7e60bc565da98eaf30ac97210db8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"aaa7e60bc565da98eaf30ac97210db8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:50.919860 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:50.919809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/02cbb65cfbfc76887ba93e94ab94587f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-112.ec2.internal\" (UID: \"02cbb65cfbfc76887ba93e94ab94587f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:50.995075 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:50.995044    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:51.020221 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:51.020191    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aaa7e60bc565da98eaf30ac97210db8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"aaa7e60bc565da98eaf30ac97210db8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:51.020277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:51.020225    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaa7e60bc565da98eaf30ac97210db8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"aaa7e60bc565da98eaf30ac97210db8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:51.020277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:51.020241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/02cbb65cfbfc76887ba93e94ab94587f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-112.ec2.internal\" (UID: \"02cbb65cfbfc76887ba93e94ab94587f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:51.020345 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:51.020287    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaa7e60bc565da98eaf30ac97210db8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"aaa7e60bc565da98eaf30ac97210db8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:51.020345 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:51.020295    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aaa7e60bc565da98eaf30ac97210db8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"aaa7e60bc565da98eaf30ac97210db8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:51.020345 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:51.020330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/02cbb65cfbfc76887ba93e94ab94587f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-112.ec2.internal\" (UID: \"02cbb65cfbfc76887ba93e94ab94587f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:51.095447 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:51.095390    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:51.181016 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:51.180976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:51.184942 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:51.184920    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:51.195884 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:51.195865    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:51.296437 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:51.296392    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:52.662455 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:52.662356    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:52.762941 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:52.762910    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:52.807261 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:52.807233    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:52.820691 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:52.820655    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 06 10:21:52.820823 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:52.820774    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 10:21:52.820869 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:52.820813    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 10:21:52.863991 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:52.863958    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:52.887705 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:52.887663    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 10:16:51 +0000 UTC" deadline="2028-05-14 00:38:10.852680934 +0000 UTC"
Sep 06 10:21:52.887705 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:52.887698    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14774h16m17.964986718s"
Sep 06 10:21:52.964675 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:52.964603    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:53.014392 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:53.014364    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cbb65cfbfc76887ba93e94ab94587f.slice/crio-ddc21ba72c7fca69848bd50b0acbc199d93a4ff86f63183c797eec215c5aaa5d WatchSource:0}: Error finding container ddc21ba72c7fca69848bd50b0acbc199d93a4ff86f63183c797eec215c5aaa5d: Status 404 returned error can't find the container with id ddc21ba72c7fca69848bd50b0acbc199d93a4ff86f63183c797eec215c5aaa5d
Sep 06 10:21:53.014640 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:53.014616    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa7e60bc565da98eaf30ac97210db8b.slice/crio-72c1ea665ddb136ba49377804767f4636916aec7b8aaf7da9d0887a49e11a2d4 WatchSource:0}: Error finding container 72c1ea665ddb136ba49377804767f4636916aec7b8aaf7da9d0887a49e11a2d4: Status 404 returned error can't find the container with id 72c1ea665ddb136ba49377804767f4636916aec7b8aaf7da9d0887a49e11a2d4
Sep 06 10:21:53.021674 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.021653    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:21:53.026202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.026160    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" event={"ID":"aaa7e60bc565da98eaf30ac97210db8b","Type":"ContainerStarted","Data":"72c1ea665ddb136ba49377804767f4636916aec7b8aaf7da9d0887a49e11a2d4"}
Sep 06 10:21:53.027402 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.027383    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" event={"ID":"02cbb65cfbfc76887ba93e94ab94587f","Type":"ContainerStarted","Data":"ddc21ba72c7fca69848bd50b0acbc199d93a4ff86f63183c797eec215c5aaa5d"}
Sep 06 10:21:53.065268 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:53.065234    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:53.165773 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:53.165742    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:53.255714 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.255639    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:53.266648 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:53.266625    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:53.367516 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:53.367486    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found"
Sep 06 10:21:53.437384 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.437349    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:53.483063 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.483024    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:53.493752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.493645    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 10:21:53.494835 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.494617    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal"
Sep 06 10:21:53.505227 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.505196    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 10:21:53.863179 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.863142    2564 apiserver.go:52] "Watching apiserver"
Sep 06 10:21:53.875907 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.875877    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 06 10:21:53.877141 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.877114    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hpfqd","kube-system/konnectivity-agent-qk4m4","openshift-cluster-node-tuning-operator/tuned-lmbfm","openshift-image-registry/node-ca-5wxtv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal","openshift-multus/network-metrics-daemon-w7hwz","openshift-network-diagnostics/network-check-target-m2fg8","openshift-ovn-kubernetes/ovnkube-node-bsdqh","kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54","openshift-dns/node-resolver-mjk44","openshift-multus/multus-additional-cni-plugins-668b9","openshift-multus/multus-hj4t9","openshift-network-operator/iptables-alerter-44hld"]
Sep 06 10:21:53.878752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.878729    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:21:53.878882 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:53.878819    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:21:53.879794 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.879773    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:53.881450 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.881431    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.882511 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.882491    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 06 10:21:53.882656 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.882636    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 06 10:21:53.882776 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.882760    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kwpll\""
Sep 06 10:21:53.882922 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.882905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:53.883617 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.883601    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 06 10:21:53.883792 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.883774    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 06 10:21:53.883856 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.883845    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:53.884080 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.884062    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6s98h\""
Sep 06 10:21:53.884226 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.884193    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 06 10:21:53.884782 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.884765    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:53.884893 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:53.884870    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:21:53.885820 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.885800    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 06 10:21:53.886442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.886423    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 06 10:21:53.886527 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.886471    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.886589 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.886555    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 06 10:21:53.886640 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.886619    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 06 10:21:53.886690 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.886638    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 06 10:21:53.886984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.886953    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 06 10:21:53.887134 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.887116    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s5krt\""
Sep 06 10:21:53.887205 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.887132    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jpqk9\""
Sep 06 10:21:53.887817 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.887795    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.888877 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.888858    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 06 10:21:53.889289 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.889102    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 06 10:21:53.889401 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.889383    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:21:53.890299 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.890280    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 06 10:21:53.890717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.890492    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 06 10:21:53.890717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.890516    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 06 10:21:53.890717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.890564    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 06 10:21:53.890894 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.890765    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 06 10:21:53.890894 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.890831    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tqvx9\""
Sep 06 10:21:53.890894 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.890879    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 06 10:21:53.891149 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.891131    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 06 10:21:53.891226 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.891185    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 06 10:21:53.891708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.891484    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fmthv\""
Sep 06 10:21:53.891708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.891607    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 06 10:21:53.891708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.891666    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 06 10:21:53.891922 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.891718    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hlscz\""
Sep 06 10:21:53.892509 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.892491    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:53.893995 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.893976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:53.894094 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:53.894075    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:21:53.894636 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.894617    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2p28f\""
Sep 06 10:21:53.894835 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.894820    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 06 10:21:53.894914 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.894868    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 06 10:21:53.894980 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.894956    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47\") pod \"network-check-target-m2fg8\" (UID: \"cc37245a-69d3-4f71-9011-40a62c5457a2\") " pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:21:53.895038 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895005    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e2d842b-52a5-4065-a84b-1e8b047e2ba7-tmp-dir\") pod \"node-resolver-mjk44\" (UID: \"7e2d842b-52a5-4065-a84b-1e8b047e2ba7\") " pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:53.895084 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895037    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-modprobe-d\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895084 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895068    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-systemd\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895179 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5b22ecb-47e8-41b9-bebd-4f17d4454220-tmp\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895179 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ad43d1d-8428-4eae-a728-b9052f618103-serviceca\") pod \"node-ca-5wxtv\" (UID: \"7ad43d1d-8428-4eae-a728-b9052f618103\") " pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:53.895420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895203    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93ce6edf-e0bf-4f03-9515-2d737ac88928-iptables-alerter-script\") pod \"iptables-alerter-44hld\" (UID: \"93ce6edf-e0bf-4f03-9515-2d737ac88928\") " pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:53.895420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895236    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 06 10:21:53.895420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e2d842b-52a5-4065-a84b-1e8b047e2ba7-hosts-file\") pod \"node-resolver-mjk44\" (UID: \"7e2d842b-52a5-4065-a84b-1e8b047e2ba7\") " pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:53.895420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:53.895420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895347    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxpm9\" (UniqueName: \"kubernetes.io/projected/7e2d842b-52a5-4065-a84b-1e8b047e2ba7-kube-api-access-gxpm9\") pod \"node-resolver-mjk44\" (UID: \"7e2d842b-52a5-4065-a84b-1e8b047e2ba7\") " pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:53.895420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895387    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-kubernetes\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-run\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-lib-modules\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895496    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-var-lib-kubelet\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ad43d1d-8428-4eae-a728-b9052f618103-host\") pod \"node-ca-5wxtv\" (UID: \"7ad43d1d-8428-4eae-a728-b9052f618103\") " pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:53.895680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895589    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-sysconfig\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895611    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-sysctl-d\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-sys\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-host\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895965 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdz4\" (UniqueName: \"kubernetes.io/projected/a5b22ecb-47e8-41b9-bebd-4f17d4454220-kube-api-access-7wdz4\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895965 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895752    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9h7\" (UniqueName: \"kubernetes.io/projected/7ad43d1d-8428-4eae-a728-b9052f618103-kube-api-access-4x9h7\") pod \"node-ca-5wxtv\" (UID: \"7ad43d1d-8428-4eae-a728-b9052f618103\") " pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:53.895965 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895783    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93ce6edf-e0bf-4f03-9515-2d737ac88928-host-slash\") pod \"iptables-alerter-44hld\" (UID: \"93ce6edf-e0bf-4f03-9515-2d737ac88928\") " pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:53.895965 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895815    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-sysctl-conf\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895965 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895843    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-tuned\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.895965 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.895865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn8kt\" (UniqueName: \"kubernetes.io/projected/93ce6edf-e0bf-4f03-9515-2d737ac88928-kube-api-access-jn8kt\") pod \"iptables-alerter-44hld\" (UID: \"93ce6edf-e0bf-4f03-9515-2d737ac88928\") " pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:53.898687 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.897882    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 06 10:21:53.898687 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.897902    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 06 10:21:53.898687 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.897920    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xtxfl\""
Sep 06 10:21:53.900272 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.899483    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 10:21:53.923630 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.923600    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hgfqg"
Sep 06 10:21:53.929328 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.929304    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hgfqg"
Sep 06 10:21:53.983804 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.983775    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 06 10:21:53.996697 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996662    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-systemd-units\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.996834 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-cni-bin\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.996834 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-var-lib-openvswitch\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.996834 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns7gm\" (UniqueName: \"kubernetes.io/projected/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-kube-api-access-ns7gm\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:53.996834 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-lib-modules\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.997009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996857    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ad43d1d-8428-4eae-a728-b9052f618103-host\") pod \"node-ca-5wxtv\" (UID: \"7ad43d1d-8428-4eae-a728-b9052f618103\") " pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:53.997009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:53.997009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-run-netns\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.997009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996938    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ad43d1d-8428-4eae-a728-b9052f618103-host\") pod \"node-ca-5wxtv\" (UID: \"7ad43d1d-8428-4eae-a728-b9052f618103\") " pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:53.997009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996948    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-kubelet-config\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:53.997009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-sysctl-d\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.997009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.996999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-host\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997023    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9h7\" (UniqueName: \"kubernetes.io/projected/7ad43d1d-8428-4eae-a728-b9052f618103-kube-api-access-4x9h7\") pod \"node-ca-5wxtv\" (UID: \"7ad43d1d-8428-4eae-a728-b9052f618103\") " pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-device-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997064    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-lib-modules\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997072    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-kubelet\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997091    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-host\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997097    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-run-openvswitch\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-run-ovn-kubernetes\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997146    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-sysctl-d\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7e1f1e1-44aa-4007-a88d-58ad1c176394-env-overrides\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997188    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn8kt\" (UniqueName: \"kubernetes.io/projected/93ce6edf-e0bf-4f03-9515-2d737ac88928-kube-api-access-jn8kt\") pod \"iptables-alerter-44hld\" (UID: \"93ce6edf-e0bf-4f03-9515-2d737ac88928\") " pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:53.997333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997228    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-cni-dir\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7e1f1e1-44aa-4007-a88d-58ad1c176394-ovnkube-script-lib\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997362    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e2d842b-52a5-4065-a84b-1e8b047e2ba7-hosts-file\") pod \"node-resolver-mjk44\" (UID: \"7e2d842b-52a5-4065-a84b-1e8b047e2ba7\") " pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997411    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-var-lib-cni-multus\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997431    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-etc-openvswitch\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-run\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997487    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-run-multus-certs\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997510    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-sys-fs\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997532    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-run\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997550    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e2d842b-52a5-4065-a84b-1e8b047e2ba7-hosts-file\") pod \"node-resolver-mjk44\" (UID: \"7e2d842b-52a5-4065-a84b-1e8b047e2ba7\") " pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997581    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-run-systemd\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-node-log\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-sysconfig\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-sys\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47\") pod \"network-check-target-m2fg8\" (UID: \"cc37245a-69d3-4f71-9011-40a62c5457a2\") " pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:21:53.997775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997691    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-cnibin\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-socket-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl529\" (UniqueName: \"kubernetes.io/projected/21ec7fd0-b053-4c24-b252-a32a59409d59-kube-api-access-zl529\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997765    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-sys\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-tuned\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-sysconfig\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997803    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e2d842b-52a5-4065-a84b-1e8b047e2ba7-tmp-dir\") pod \"node-resolver-mjk44\" (UID: \"7e2d842b-52a5-4065-a84b-1e8b047e2ba7\") " pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997862    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-socket-dir-parent\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997911    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-daemon-config\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997936    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7e1f1e1-44aa-4007-a88d-58ad1c176394-ovn-node-metrics-cert\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997956    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-dbus\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.997981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-modprobe-d\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998018    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-systemd\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5b22ecb-47e8-41b9-bebd-4f17d4454220-tmp\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-systemd\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93ce6edf-e0bf-4f03-9515-2d737ac88928-iptables-alerter-script\") pod \"iptables-alerter-44hld\" (UID: \"93ce6edf-e0bf-4f03-9515-2d737ac88928\") " pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:53.998453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998120    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-modprobe-d\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998129    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-system-cni-dir\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e2d842b-52a5-4065-a84b-1e8b047e2ba7-tmp-dir\") pod \"node-resolver-mjk44\" (UID: \"7e2d842b-52a5-4065-a84b-1e8b047e2ba7\") " pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998170    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-conf-dir\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998192    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-slash\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998243    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-var-lib-kubelet\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-run-netns\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-var-lib-kubelet\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998380    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-run-ovn\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998422    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998450    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93ce6edf-e0bf-4f03-9515-2d737ac88928-host-slash\") pod \"iptables-alerter-44hld\" (UID: \"93ce6edf-e0bf-4f03-9515-2d737ac88928\") " pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-cni-binary-copy\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998488    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93ce6edf-e0bf-4f03-9515-2d737ac88928-host-slash\") pod \"iptables-alerter-44hld\" (UID: \"93ce6edf-e0bf-4f03-9515-2d737ac88928\") " pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-run-k8s-cni-cncf-io\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998531    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-var-lib-cni-bin\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b01a3bd2-a82b-43e9-81b7-f94eeede7fd5-konnectivity-ca\") pod \"konnectivity-agent-qk4m4\" (UID: \"b01a3bd2-a82b-43e9-81b7-f94eeede7fd5\") " pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:21:53.999183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998551    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-log-socket\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998618    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-cni-netd\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7e1f1e1-44aa-4007-a88d-58ad1c176394-ovnkube-config\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998663    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93ce6edf-e0bf-4f03-9515-2d737ac88928-iptables-alerter-script\") pod \"iptables-alerter-44hld\" (UID: \"93ce6edf-e0bf-4f03-9515-2d737ac88928\") " pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998665    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-sysctl-conf\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-system-cni-dir\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998745    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-cnibin\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998756    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-sysctl-conf\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ad43d1d-8428-4eae-a728-b9052f618103-serviceca\") pod \"node-ca-5wxtv\" (UID: \"7ad43d1d-8428-4eae-a728-b9052f618103\") " pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998808    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxpm9\" (UniqueName: \"kubernetes.io/projected/7e2d842b-52a5-4065-a84b-1e8b047e2ba7-kube-api-access-gxpm9\") pod \"node-resolver-mjk44\" (UID: \"7e2d842b-52a5-4065-a84b-1e8b047e2ba7\") " pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998901    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-var-lib-kubelet\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.998991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-etc-kubernetes\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999060    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b01a3bd2-a82b-43e9-81b7-f94eeede7fd5-agent-certs\") pod \"konnectivity-agent-qk4m4\" (UID: \"b01a3bd2-a82b-43e9-81b7-f94eeede7fd5\") " pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999184    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwv6w\" (UniqueName: \"kubernetes.io/projected/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-kube-api-access-bwv6w\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ad43d1d-8428-4eae-a728-b9052f618103-serviceca\") pod \"node-ca-5wxtv\" (UID: \"7ad43d1d-8428-4eae-a728-b9052f618103\") " pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999236    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-cni-binary-copy\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:53.999891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-kubernetes\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:54.000522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999286    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-hostroot\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.000522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999327    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99ns\" (UniqueName: \"kubernetes.io/projected/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-kube-api-access-p99ns\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.000522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-registration-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.000522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999374    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-kubernetes\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:54.000522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:54.000522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999439    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.000522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdz4\" (UniqueName: \"kubernetes.io/projected/a5b22ecb-47e8-41b9-bebd-4f17d4454220-kube-api-access-7wdz4\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:54.000522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:53.999489    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghwv\" (UniqueName: \"kubernetes.io/projected/c7e1f1e1-44aa-4007-a88d-58ad1c176394-kube-api-access-2ghwv\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.002064 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.002045    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5b22ecb-47e8-41b9-bebd-4f17d4454220-tmp\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:54.002156 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.002085    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a5b22ecb-47e8-41b9-bebd-4f17d4454220-etc-tuned\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:54.004719 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.004697    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:21:54.004812 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.004723    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:21:54.004812 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.004735    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bst47 for pod openshift-network-diagnostics/network-check-target-m2fg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:54.004812 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.004795    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47 podName:cc37245a-69d3-4f71-9011-40a62c5457a2 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:54.504773483 +0000 UTC m=+2.988273581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bst47" (UniqueName: "kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47") pod "network-check-target-m2fg8" (UID: "cc37245a-69d3-4f71-9011-40a62c5457a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:54.006597 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.006526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9h7\" (UniqueName: \"kubernetes.io/projected/7ad43d1d-8428-4eae-a728-b9052f618103-kube-api-access-4x9h7\") pod \"node-ca-5wxtv\" (UID: \"7ad43d1d-8428-4eae-a728-b9052f618103\") " pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:54.006704 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.006675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn8kt\" (UniqueName: \"kubernetes.io/projected/93ce6edf-e0bf-4f03-9515-2d737ac88928-kube-api-access-jn8kt\") pod \"iptables-alerter-44hld\" (UID: \"93ce6edf-e0bf-4f03-9515-2d737ac88928\") " pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:54.008356 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.008332    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxpm9\" (UniqueName: \"kubernetes.io/projected/7e2d842b-52a5-4065-a84b-1e8b047e2ba7-kube-api-access-gxpm9\") pod \"node-resolver-mjk44\" (UID: \"7e2d842b-52a5-4065-a84b-1e8b047e2ba7\") " pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:54.008807 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.008787    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdz4\" (UniqueName: \"kubernetes.io/projected/a5b22ecb-47e8-41b9-bebd-4f17d4454220-kube-api-access-7wdz4\") pod \"tuned-lmbfm\" (UID: \"a5b22ecb-47e8-41b9-bebd-4f17d4454220\") " pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:54.100133 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.100133 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-run-netns\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100160    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-kubelet-config\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-device-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100203    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-kubelet\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-run-openvswitch\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-run-netns\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100256    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-run-ovn-kubernetes\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100311    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7e1f1e1-44aa-4007-a88d-58ad1c176394-env-overrides\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100341    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-kubelet\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-cni-dir\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-kubelet-config\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-run-openvswitch\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.100385 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100383    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-run-ovn-kubernetes\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100393    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7e1f1e1-44aa-4007-a88d-58ad1c176394-ovnkube-script-lib\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100387    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-device-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100406    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-cni-dir\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100460    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-var-lib-cni-multus\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-etc-openvswitch\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100499    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-var-lib-cni-multus\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-run-multus-certs\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-run-multus-certs\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-sys-fs\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100563    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-etc-openvswitch\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-run-systemd\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100629    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-sys-fs\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-run-systemd\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.100694    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:54.101029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-node-log\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.100769    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs podName:21ec7fd0-b053-4c24-b252-a32a59409d59 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:54.600753724 +0000 UTC m=+3.084253830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs") pod "network-metrics-daemon-w7hwz" (UID: "21ec7fd0-b053-4c24-b252-a32a59409d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-cnibin\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100765    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-node-log\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100818    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7e1f1e1-44aa-4007-a88d-58ad1c176394-env-overrides\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-socket-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100859    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-cnibin\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl529\" (UniqueName: \"kubernetes.io/projected/21ec7fd0-b053-4c24-b252-a32a59409d59-kube-api-access-zl529\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100910    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7e1f1e1-44aa-4007-a88d-58ad1c176394-ovnkube-script-lib\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-socket-dir-parent\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100938    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-socket-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100953    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-socket-dir-parent\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100958    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-daemon-config\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.100988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7e1f1e1-44aa-4007-a88d-58ad1c176394-ovn-node-metrics-cert\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-dbus\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:54.101859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-system-cni-dir\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101070    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-dbus\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-conf-dir\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-slash\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-system-cni-dir\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-conf-dir\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101154    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-slash\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-run-netns\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-run-ovn\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101193    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-run-netns\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-cni-binary-copy\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-run-ovn\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-run-k8s-cni-cncf-io\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-var-lib-cni-bin\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101325    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-run-k8s-cni-cncf-io\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.102706 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101358    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b01a3bd2-a82b-43e9-81b7-f94eeede7fd5-konnectivity-ca\") pod \"konnectivity-agent-qk4m4\" (UID: \"b01a3bd2-a82b-43e9-81b7-f94eeede7fd5\") " pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-var-lib-cni-bin\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-log-socket\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-cni-netd\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7e1f1e1-44aa-4007-a88d-58ad1c176394-ovnkube-config\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101455    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-cni-netd\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-log-socket\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101466    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-system-cni-dir\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101455    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-multus-daemon-config\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101492    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-cnibin\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101517    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-system-cni-dir\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101517    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-var-lib-kubelet\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-etc-kubernetes\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101548    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-host-var-lib-kubelet\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b01a3bd2-a82b-43e9-81b7-f94eeede7fd5-agent-certs\") pod \"konnectivity-agent-qk4m4\" (UID: \"b01a3bd2-a82b-43e9-81b7-f94eeede7fd5\") " pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101599    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwv6w\" (UniqueName: \"kubernetes.io/projected/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-kube-api-access-bwv6w\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101612    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-cnibin\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.103553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-cni-binary-copy\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-hostroot\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99ns\" (UniqueName: \"kubernetes.io/projected/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-kube-api-access-p99ns\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101705    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-registration-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101723    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101758    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101781    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-etc-kubernetes\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghwv\" (UniqueName: \"kubernetes.io/projected/c7e1f1e1-44aa-4007-a88d-58ad1c176394-kube-api-access-2ghwv\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101808    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-cni-binary-copy\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-registration-dir\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.101918    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101936    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b01a3bd2-a82b-43e9-81b7-f94eeede7fd5-konnectivity-ca\") pod \"konnectivity-agent-qk4m4\" (UID: \"b01a3bd2-a82b-43e9-81b7-f94eeede7fd5\") " pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.101953    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7e1f1e1-44aa-4007-a88d-58ad1c176394-ovnkube-config\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.101994    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret podName:6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d nodeName:}" failed. No retries permitted until 2026-09-06 10:21:54.601974815 +0000 UTC m=+3.085474915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret") pod "global-pull-secret-syncer-hpfqd" (UID: "6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.102043    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-hostroot\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.102073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-systemd-units\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.104260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.102108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-cni-bin\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.105006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.102131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-var-lib-openvswitch\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.105006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.102164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns7gm\" (UniqueName: \"kubernetes.io/projected/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-kube-api-access-ns7gm\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.105006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.102241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-host-cni-bin\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.105006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.102281    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-systemd-units\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.105006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.102330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7e1f1e1-44aa-4007-a88d-58ad1c176394-var-lib-openvswitch\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.105006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.102344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-cni-binary-copy\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.105006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.102440    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.105006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.103575    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7e1f1e1-44aa-4007-a88d-58ad1c176394-ovn-node-metrics-cert\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.105006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.104123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b01a3bd2-a82b-43e9-81b7-f94eeede7fd5-agent-certs\") pod \"konnectivity-agent-qk4m4\" (UID: \"b01a3bd2-a82b-43e9-81b7-f94eeede7fd5\") " pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:21:54.112954 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.112921    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl529\" (UniqueName: \"kubernetes.io/projected/21ec7fd0-b053-4c24-b252-a32a59409d59-kube-api-access-zl529\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:54.113447 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.113257    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwv6w\" (UniqueName: \"kubernetes.io/projected/f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a-kube-api-access-bwv6w\") pod \"aws-ebs-csi-driver-node-xkd54\" (UID: \"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.113447 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.113342    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99ns\" (UniqueName: \"kubernetes.io/projected/198a5f6a-7c03-42ab-9a85-1efd3f6e958a-kube-api-access-p99ns\") pod \"multus-hj4t9\" (UID: \"198a5f6a-7c03-42ab-9a85-1efd3f6e958a\") " pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.113620 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.113492    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns7gm\" (UniqueName: \"kubernetes.io/projected/7fa465d8-bbd3-4102-ae4f-7bba5685b27d-kube-api-access-ns7gm\") pod \"multus-additional-cni-plugins-668b9\" (UID: \"7fa465d8-bbd3-4102-ae4f-7bba5685b27d\") " pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.113667 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.113613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghwv\" (UniqueName: \"kubernetes.io/projected/c7e1f1e1-44aa-4007-a88d-58ad1c176394-kube-api-access-2ghwv\") pod \"ovnkube-node-bsdqh\" (UID: \"c7e1f1e1-44aa-4007-a88d-58ad1c176394\") " pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.192492 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.192461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mjk44"
Sep 06 10:21:54.202291 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.202267    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lmbfm"
Sep 06 10:21:54.203957 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.203849    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:54.210041 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.210016    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54"
Sep 06 10:21:54.214901 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.214881    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5wxtv"
Sep 06 10:21:54.221517 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.221499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44hld"
Sep 06 10:21:54.228252 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.228233    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:21:54.234954 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.234936    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hj4t9"
Sep 06 10:21:54.240451 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.240425    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:21:54.244179 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.244161    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-668b9"
Sep 06 10:21:54.505027 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.504951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47\") pod \"network-check-target-m2fg8\" (UID: \"cc37245a-69d3-4f71-9011-40a62c5457a2\") " pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:21:54.505197 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.505138    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:21:54.505197 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.505163    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:21:54.505197 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.505176    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bst47 for pod openshift-network-diagnostics/network-check-target-m2fg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:54.505375 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.505260    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47 podName:cc37245a-69d3-4f71-9011-40a62c5457a2 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:55.505239237 +0000 UTC m=+3.988739327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bst47" (UniqueName: "kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47") pod "network-check-target-m2fg8" (UID: "cc37245a-69d3-4f71-9011-40a62c5457a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:54.606180 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.606146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:54.606338 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.606199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:54.606338 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.606325    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:54.606438 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.606360    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:54.606438 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.606401    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret podName:6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d nodeName:}" failed. No retries permitted until 2026-09-06 10:21:55.606379952 +0000 UTC m=+4.089880041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret") pod "global-pull-secret-syncer-hpfqd" (UID: "6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:54.606438 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:54.606420    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs podName:21ec7fd0-b053-4c24-b252-a32a59409d59 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:55.606410358 +0000 UTC m=+4.089910453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs") pod "network-metrics-daemon-w7hwz" (UID: "21ec7fd0-b053-4c24-b252-a32a59409d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:54.817339 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:54.817130    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad43d1d_8428_4eae_a728_b9052f618103.slice/crio-cfdb4aefb9146d2e04e8412811c8cb88af19f44d423a7f1fec590d54f24031b4 WatchSource:0}: Error finding container cfdb4aefb9146d2e04e8412811c8cb88af19f44d423a7f1fec590d54f24031b4: Status 404 returned error can't find the container with id cfdb4aefb9146d2e04e8412811c8cb88af19f44d423a7f1fec590d54f24031b4
Sep 06 10:21:54.820195 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:54.820167    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2d842b_52a5_4065_a84b_1e8b047e2ba7.slice/crio-172b6d670e77ea64e1dad8eb6185e0125b256c4fed933bc65306a5b1e63368ed WatchSource:0}: Error finding container 172b6d670e77ea64e1dad8eb6185e0125b256c4fed933bc65306a5b1e63368ed: Status 404 returned error can't find the container with id 172b6d670e77ea64e1dad8eb6185e0125b256c4fed933bc65306a5b1e63368ed
Sep 06 10:21:54.821452 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:54.821427    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ce6edf_e0bf_4f03_9515_2d737ac88928.slice/crio-fdfbd056d5e151b0dfd3097d9fa3b96d0d80a5cafa2c52cde10685a8143b9cf2 WatchSource:0}: Error finding container fdfbd056d5e151b0dfd3097d9fa3b96d0d80a5cafa2c52cde10685a8143b9cf2: Status 404 returned error can't find the container with id fdfbd056d5e151b0dfd3097d9fa3b96d0d80a5cafa2c52cde10685a8143b9cf2
Sep 06 10:21:54.822612 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:54.822586    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198a5f6a_7c03_42ab_9a85_1efd3f6e958a.slice/crio-886ac08ab89f6632fd9be670e70ef93f7b91a206004b4ef73604f980d0e859cb WatchSource:0}: Error finding container 886ac08ab89f6632fd9be670e70ef93f7b91a206004b4ef73604f980d0e859cb: Status 404 returned error can't find the container with id 886ac08ab89f6632fd9be670e70ef93f7b91a206004b4ef73604f980d0e859cb
Sep 06 10:21:54.824442 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:54.824412    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b22ecb_47e8_41b9_bebd_4f17d4454220.slice/crio-668ac65e47548f4232cf7bbe63efc1a0a01d6e8e3ede4f02bbd8d341644c32a4 WatchSource:0}: Error finding container 668ac65e47548f4232cf7bbe63efc1a0a01d6e8e3ede4f02bbd8d341644c32a4: Status 404 returned error can't find the container with id 668ac65e47548f4232cf7bbe63efc1a0a01d6e8e3ede4f02bbd8d341644c32a4
Sep 06 10:21:54.825957 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:54.825937    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01a3bd2_a82b_43e9_81b7_f94eeede7fd5.slice/crio-9c9f7ef898c17158b855b01224349c1cbf171f8a46aea92f1e7bd9fb1783473c WatchSource:0}: Error finding container 9c9f7ef898c17158b855b01224349c1cbf171f8a46aea92f1e7bd9fb1783473c: Status 404 returned error can't find the container with id 9c9f7ef898c17158b855b01224349c1cbf171f8a46aea92f1e7bd9fb1783473c
Sep 06 10:21:54.826744 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:54.826721    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e1f1e1_44aa_4007_a88d_58ad1c176394.slice/crio-1a90b5099fbd8f8059be8f7b522ea3b92bcade3786a17a63e2cdda525ee73cb9 WatchSource:0}: Error finding container 1a90b5099fbd8f8059be8f7b522ea3b92bcade3786a17a63e2cdda525ee73cb9: Status 404 returned error can't find the container with id 1a90b5099fbd8f8059be8f7b522ea3b92bcade3786a17a63e2cdda525ee73cb9
Sep 06 10:21:54.827612 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:54.827591    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa465d8_bbd3_4102_ae4f_7bba5685b27d.slice/crio-2fd8bacb23ae3f23677dbef5408f2b5ea957158b7af0fe509367a39b836abe05 WatchSource:0}: Error finding container 2fd8bacb23ae3f23677dbef5408f2b5ea957158b7af0fe509367a39b836abe05: Status 404 returned error can't find the container with id 2fd8bacb23ae3f23677dbef5408f2b5ea957158b7af0fe509367a39b836abe05
Sep 06 10:21:54.828377 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:21:54.828358    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c42e73_def7_412e_8dd2_4d9c9ad7ce8a.slice/crio-c7e13fb21e3fd6053e810a89a15b0b88c911539be6b8985af44dfd540f497ff4 WatchSource:0}: Error finding container c7e13fb21e3fd6053e810a89a15b0b88c911539be6b8985af44dfd540f497ff4: Status 404 returned error can't find the container with id c7e13fb21e3fd6053e810a89a15b0b88c911539be6b8985af44dfd540f497ff4
Sep 06 10:21:54.931697 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.931663    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 10:16:53 +0000 UTC" deadline="2028-03-08 08:46:17.415780696 +0000 UTC"
Sep 06 10:21:54.931697 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:54.931691    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13174h24m22.484092489s"
Sep 06 10:21:55.031230 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.031179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" event={"ID":"02cbb65cfbfc76887ba93e94ab94587f","Type":"ContainerStarted","Data":"43f8d748fc4ef3cb809e04c782ac44f5376408dcdda027c0cf5714bee0fecf23"}
Sep 06 10:21:55.032499 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.032475    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-668b9" event={"ID":"7fa465d8-bbd3-4102-ae4f-7bba5685b27d","Type":"ContainerStarted","Data":"2fd8bacb23ae3f23677dbef5408f2b5ea957158b7af0fe509367a39b836abe05"}
Sep 06 10:21:55.033532 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.033511    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" event={"ID":"c7e1f1e1-44aa-4007-a88d-58ad1c176394","Type":"ContainerStarted","Data":"1a90b5099fbd8f8059be8f7b522ea3b92bcade3786a17a63e2cdda525ee73cb9"}
Sep 06 10:21:55.035040 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.035015    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lmbfm" event={"ID":"a5b22ecb-47e8-41b9-bebd-4f17d4454220","Type":"ContainerStarted","Data":"668ac65e47548f4232cf7bbe63efc1a0a01d6e8e3ede4f02bbd8d341644c32a4"}
Sep 06 10:21:55.036133 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.036116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44hld" event={"ID":"93ce6edf-e0bf-4f03-9515-2d737ac88928","Type":"ContainerStarted","Data":"fdfbd056d5e151b0dfd3097d9fa3b96d0d80a5cafa2c52cde10685a8143b9cf2"}
Sep 06 10:21:55.037176 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.037159    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hj4t9" event={"ID":"198a5f6a-7c03-42ab-9a85-1efd3f6e958a","Type":"ContainerStarted","Data":"886ac08ab89f6632fd9be670e70ef93f7b91a206004b4ef73604f980d0e859cb"}
Sep 06 10:21:55.038071 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.038054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54" event={"ID":"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a","Type":"ContainerStarted","Data":"c7e13fb21e3fd6053e810a89a15b0b88c911539be6b8985af44dfd540f497ff4"}
Sep 06 10:21:55.039040 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.039022    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qk4m4" event={"ID":"b01a3bd2-a82b-43e9-81b7-f94eeede7fd5","Type":"ContainerStarted","Data":"9c9f7ef898c17158b855b01224349c1cbf171f8a46aea92f1e7bd9fb1783473c"}
Sep 06 10:21:55.042481 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.040378    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mjk44" event={"ID":"7e2d842b-52a5-4065-a84b-1e8b047e2ba7","Type":"ContainerStarted","Data":"172b6d670e77ea64e1dad8eb6185e0125b256c4fed933bc65306a5b1e63368ed"}
Sep 06 10:21:55.044051 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.044028    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5wxtv" event={"ID":"7ad43d1d-8428-4eae-a728-b9052f618103","Type":"ContainerStarted","Data":"cfdb4aefb9146d2e04e8412811c8cb88af19f44d423a7f1fec590d54f24031b4"}
Sep 06 10:21:55.047069 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.047032    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" podStartSLOduration=2.047021419 podStartE2EDuration="2.047021419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:21:55.046798869 +0000 UTC m=+3.530298978" watchObservedRunningTime="2026-09-06 10:21:55.047021419 +0000 UTC m=+3.530521528"
Sep 06 10:21:55.515156 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.515027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47\") pod \"network-check-target-m2fg8\" (UID: \"cc37245a-69d3-4f71-9011-40a62c5457a2\") " pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:21:55.515345 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:55.515301    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:21:55.515345 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:55.515326    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:21:55.515345 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:55.515340    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bst47 for pod openshift-network-diagnostics/network-check-target-m2fg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:55.515516 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:55.515403    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47 podName:cc37245a-69d3-4f71-9011-40a62c5457a2 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:57.515384007 +0000 UTC m=+5.998884101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bst47" (UniqueName: "kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47") pod "network-check-target-m2fg8" (UID: "cc37245a-69d3-4f71-9011-40a62c5457a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:55.616595 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.616252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:55.616595 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.616321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:55.616595 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:55.616488    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:55.616595 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:55.616552    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs podName:21ec7fd0-b053-4c24-b252-a32a59409d59 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:57.616532844 +0000 UTC m=+6.100032935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs") pod "network-metrics-daemon-w7hwz" (UID: "21ec7fd0-b053-4c24-b252-a32a59409d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:55.617000 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:55.616942    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:55.617000 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:55.616989    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret podName:6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d nodeName:}" failed. No retries permitted until 2026-09-06 10:21:57.616973885 +0000 UTC m=+6.100473972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret") pod "global-pull-secret-syncer-hpfqd" (UID: "6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:55.880506 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.880471    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:55.932913 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.932849    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 10:16:53 +0000 UTC" deadline="2028-05-13 23:18:08.227540548 +0000 UTC"
Sep 06 10:21:55.932913 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:55.932910    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14772h56m12.294634336s"
Sep 06 10:21:56.023334 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:56.023302    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:56.023503 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:56.023435    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:21:56.023853 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:56.023831    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:21:56.023942 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:56.023930    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:21:56.026322 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:56.026298    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:56.026434 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:56.026405    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:21:56.055987 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:56.055954    2564 generic.go:356] "Generic (PLEG): container finished" podID="aaa7e60bc565da98eaf30ac97210db8b" containerID="d293ad9cfe306f3100d8e400006bd8347fa407c72c4709438ada85390ae94761" exitCode=0
Sep 06 10:21:56.056238 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:56.056063    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" event={"ID":"aaa7e60bc565da98eaf30ac97210db8b","Type":"ContainerDied","Data":"d293ad9cfe306f3100d8e400006bd8347fa407c72c4709438ada85390ae94761"}
Sep 06 10:21:57.072239 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:57.071538    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" event={"ID":"aaa7e60bc565da98eaf30ac97210db8b","Type":"ContainerStarted","Data":"e8966acbdf50c586f7916af9e5e07412f3b30ace466830444851f947e6ab040c"}
Sep 06 10:21:57.533181 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:57.533143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47\") pod \"network-check-target-m2fg8\" (UID: \"cc37245a-69d3-4f71-9011-40a62c5457a2\") " pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:21:57.533374 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:57.533324    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:21:57.533374 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:57.533344    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:21:57.533374 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:57.533355    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bst47 for pod openshift-network-diagnostics/network-check-target-m2fg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:57.533530 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:57.533415    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47 podName:cc37245a-69d3-4f71-9011-40a62c5457a2 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:01.533396518 +0000 UTC m=+10.016896628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bst47" (UniqueName: "kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47") pod "network-check-target-m2fg8" (UID: "cc37245a-69d3-4f71-9011-40a62c5457a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:57.634382 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:57.634344    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:57.634552 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:57.634434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:57.634622 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:57.634558    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:57.634671 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:57.634623    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret podName:6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d nodeName:}" failed. No retries permitted until 2026-09-06 10:22:01.63460378 +0000 UTC m=+10.118103871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret") pod "global-pull-secret-syncer-hpfqd" (UID: "6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:57.635073 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:57.635051    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:57.635186 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:57.635106    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs podName:21ec7fd0-b053-4c24-b252-a32a59409d59 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:01.635091528 +0000 UTC m=+10.118591629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs") pod "network-metrics-daemon-w7hwz" (UID: "21ec7fd0-b053-4c24-b252-a32a59409d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:58.023113 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:58.022712    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:21:58.023113 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:58.022843    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:21:58.023113 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:58.022876    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:21:58.023113 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:21:58.023031    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:21:58.023113 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:58.023064    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:21:58.023492 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:21:58.023129    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:00.023473 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:00.022973    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:00.023473 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:00.023120    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:00.023473 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:00.023122    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:00.023473 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:00.023148    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:00.023473 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:00.023239    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:00.023473 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:00.023315    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:01.569588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:01.569551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47\") pod \"network-check-target-m2fg8\" (UID: \"cc37245a-69d3-4f71-9011-40a62c5457a2\") " pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:01.570086 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:01.569739    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:22:01.570086 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:01.569757    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:22:01.570086 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:01.569770    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bst47 for pod openshift-network-diagnostics/network-check-target-m2fg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:22:01.570086 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:01.569827    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47 podName:cc37245a-69d3-4f71-9011-40a62c5457a2 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:09.569811446 +0000 UTC m=+18.053311533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bst47" (UniqueName: "kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47") pod "network-check-target-m2fg8" (UID: "cc37245a-69d3-4f71-9011-40a62c5457a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:22:01.670925 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:01.670885    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:01.671083 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:01.670977    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:01.671083 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:01.671051    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:22:01.671203 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:01.671088    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:22:01.671203 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:01.671118    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs podName:21ec7fd0-b053-4c24-b252-a32a59409d59 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:09.671101635 +0000 UTC m=+18.154601722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs") pod "network-metrics-daemon-w7hwz" (UID: "21ec7fd0-b053-4c24-b252-a32a59409d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:22:01.671203 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:01.671140    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret podName:6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d nodeName:}" failed. No retries permitted until 2026-09-06 10:22:09.671124595 +0000 UTC m=+18.154624689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret") pod "global-pull-secret-syncer-hpfqd" (UID: "6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:22:02.022996 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:02.022853    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:02.022996 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:02.022867    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:02.024036 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:02.024006    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:02.024165 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:02.024114    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:02.024398 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:02.024368    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:02.024574 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:02.024551    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:04.022628 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:04.022451    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:04.023069 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:04.022451    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:04.023069 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:04.022700    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:04.023069 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:04.022463    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:04.023069 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:04.022817    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:04.023069 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:04.022897    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:06.022914 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:06.022874    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:06.023398 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:06.022875    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:06.023398 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:06.023040    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:06.023398 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:06.022881    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:06.023398 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:06.023089    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:06.023398 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:06.023125    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:08.023145 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:08.023105    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:08.023623 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:08.023105    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:08.023623 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:08.023243    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:08.023623 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:08.023257    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:08.023623 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:08.023345    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:08.023623 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:08.023402    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:09.624104 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:09.624064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47\") pod \"network-check-target-m2fg8\" (UID: \"cc37245a-69d3-4f71-9011-40a62c5457a2\") " pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:09.624586 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:09.624241    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:22:09.624586 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:09.624266    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:22:09.624586 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:09.624280    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bst47 for pod openshift-network-diagnostics/network-check-target-m2fg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:22:09.624586 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:09.624341    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47 podName:cc37245a-69d3-4f71-9011-40a62c5457a2 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:25.624326112 +0000 UTC m=+34.107826199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bst47" (UniqueName: "kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47") pod "network-check-target-m2fg8" (UID: "cc37245a-69d3-4f71-9011-40a62c5457a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:22:09.725282 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:09.725247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:09.725450 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:09.725308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:09.725450 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:09.725408    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:22:09.725578 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:09.725483    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret podName:6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d nodeName:}" failed. No retries permitted until 2026-09-06 10:22:25.725466726 +0000 UTC m=+34.208966813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret") pod "global-pull-secret-syncer-hpfqd" (UID: "6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:22:09.725578 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:09.725414    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:22:09.725578 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:09.725568    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs podName:21ec7fd0-b053-4c24-b252-a32a59409d59 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:25.725549111 +0000 UTC m=+34.209049205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs") pod "network-metrics-daemon-w7hwz" (UID: "21ec7fd0-b053-4c24-b252-a32a59409d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:22:10.022831 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:10.022755    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:10.023011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:10.022760    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:10.023011 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:10.022893    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:10.023011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:10.022755    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:10.023011 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:10.022965    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:10.023201 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:10.023043    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:12.023947 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:12.023906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:12.024382 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:12.024027    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:12.024382 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:12.024055    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:12.024382 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:12.024025    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:12.024382 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:12.024140    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:12.024382 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:12.024259    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:13.110075 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.109645    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mjk44" event={"ID":"7e2d842b-52a5-4065-a84b-1e8b047e2ba7","Type":"ContainerStarted","Data":"62fa1500c7d7ff73aa788432df92dc7dae4d77f3f0cdc072643d3d36b928a75e"}
Sep 06 10:22:13.113107 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.113055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5wxtv" event={"ID":"7ad43d1d-8428-4eae-a728-b9052f618103","Type":"ContainerStarted","Data":"f045bcfbeae83b8c4723f3c267dc992c2e890fa88038c93803f0c374c9dae02f"}
Sep 06 10:22:13.115660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.115427    2564 generic.go:356] "Generic (PLEG): container finished" podID="7fa465d8-bbd3-4102-ae4f-7bba5685b27d" containerID="28b4c01ad0106c758892d9bceec7965a6430bbc48653c043d3354025ec9ce8ef" exitCode=0
Sep 06 10:22:13.115660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.115523    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-668b9" event={"ID":"7fa465d8-bbd3-4102-ae4f-7bba5685b27d","Type":"ContainerDied","Data":"28b4c01ad0106c758892d9bceec7965a6430bbc48653c043d3354025ec9ce8ef"}
Sep 06 10:22:13.121166 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.121144    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:22:13.122103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.121688    2564 generic.go:356] "Generic (PLEG): container finished" podID="c7e1f1e1-44aa-4007-a88d-58ad1c176394" containerID="b40b37ce831e80349a9bf00e60055196e3590eb25f795e8e91c329a58111f94a" exitCode=1
Sep 06 10:22:13.122103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.121775    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" event={"ID":"c7e1f1e1-44aa-4007-a88d-58ad1c176394","Type":"ContainerStarted","Data":"b9399e9a7f36a6287595f0ac76a9e7a6ced171bc98278364d204abe17b01b0ea"}
Sep 06 10:22:13.122103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.121854    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" event={"ID":"c7e1f1e1-44aa-4007-a88d-58ad1c176394","Type":"ContainerStarted","Data":"c0246feadc696d96d91c429d8509ce8cbe72c7262bc483f9542fb0143f41b8c2"}
Sep 06 10:22:13.122103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.121923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" event={"ID":"c7e1f1e1-44aa-4007-a88d-58ad1c176394","Type":"ContainerStarted","Data":"284f392add2652c802c043b718ad3ea4d58e50110f871305b7729f72b05dc95c"}
Sep 06 10:22:13.122103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.121976    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" event={"ID":"c7e1f1e1-44aa-4007-a88d-58ad1c176394","Type":"ContainerDied","Data":"b40b37ce831e80349a9bf00e60055196e3590eb25f795e8e91c329a58111f94a"}
Sep 06 10:22:13.122103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.122031    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" event={"ID":"c7e1f1e1-44aa-4007-a88d-58ad1c176394","Type":"ContainerStarted","Data":"3e31320fbc1e3b852c05a7a81555298b61672ad33d4a4e9e7f40858f43ae1616"}
Sep 06 10:22:13.124291 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.124262    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lmbfm" event={"ID":"a5b22ecb-47e8-41b9-bebd-4f17d4454220","Type":"ContainerStarted","Data":"9042af3d5a94172ef1e00e8516ea82511d785382e9e0af896d76ad7719f88576"}
Sep 06 10:22:13.127482 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.126551    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" podStartSLOduration=20.126535663 podStartE2EDuration="20.126535663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:21:57.088353309 +0000 UTC m=+5.571853419" watchObservedRunningTime="2026-09-06 10:22:13.126535663 +0000 UTC m=+21.610035773"
Sep 06 10:22:13.127482 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.127000    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mjk44" podStartSLOduration=3.5298002779999997 podStartE2EDuration="21.126983406s" totalImagesPullingTime="17.597183128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:13.126013301 +0000 UTC m=+21.609513434" watchObservedRunningTime="2026-09-06 10:22:13.126983406 +0000 UTC m=+21.610483517"
Sep 06 10:22:13.130808 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.130784    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hj4t9" event={"ID":"198a5f6a-7c03-42ab-9a85-1efd3f6e958a","Type":"ContainerStarted","Data":"effea89c426b3d4003343d4c4ea2a11bfadf8d002b4d0fcc9d5a8647bb9e16c5"}
Sep 06 10:22:13.132983 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.132959    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54" event={"ID":"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a","Type":"ContainerStarted","Data":"78cb5b0896f1f2bff7c4537c6ee97f4348fc02f2ac68b7b239ff02c3c61d3488"}
Sep 06 10:22:13.135495 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.135473    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qk4m4" event={"ID":"b01a3bd2-a82b-43e9-81b7-f94eeede7fd5","Type":"ContainerStarted","Data":"5c968df9a643a02fc5fa71c474058bd13bcd1a84d99db1ba7bde4e17e52604bb"}
Sep 06 10:22:13.141568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.141529    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lmbfm" podStartSLOduration=3.547970077 podStartE2EDuration="21.141516321s" totalImagesPullingTime="17.593546244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:13.141125482 +0000 UTC m=+21.624625618" watchObservedRunningTime="2026-09-06 10:22:13.141516321 +0000 UTC m=+21.625016430"
Sep 06 10:22:13.155271 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.155201    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5wxtv" podStartSLOduration=3.554401307 podStartE2EDuration="21.155187599s" totalImagesPullingTime="17.600786292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:13.154724661 +0000 UTC m=+21.638224973" watchObservedRunningTime="2026-09-06 10:22:13.155187599 +0000 UTC m=+21.638687707"
Sep 06 10:22:13.190945 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.190888    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hj4t9" podStartSLOduration=3.097415963 podStartE2EDuration="21.190868034s" totalImagesPullingTime="18.093452071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:13.190624874 +0000 UTC m=+21.674124984" watchObservedRunningTime="2026-09-06 10:22:13.190868034 +0000 UTC m=+21.674368144"
Sep 06 10:22:13.205643 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.205462    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qk4m4" podStartSLOduration=10.953364613 podStartE2EDuration="21.205439982s" totalImagesPullingTime="10.252075369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:13.204747488 +0000 UTC m=+21.688247626" watchObservedRunningTime="2026-09-06 10:22:13.205439982 +0000 UTC m=+21.688940092"
Sep 06 10:22:13.925677 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.925510    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 06 10:22:13.947420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.947309    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T10:22:13.925672422Z","UUID":"a276b47f-a34d-46b0-9dfc-a2a82717b9dc","Handler":null,"Name":"","Endpoint":""}
Sep 06 10:22:13.949275 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.949238    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 06 10:22:13.949381 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.949297    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 06 10:22:13.992017 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.991939    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:22:13.992528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:13.992511    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:22:14.022379 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:14.022353    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:14.022516 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:14.022353    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:14.022516 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:14.022477    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:14.022627 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:14.022357    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:14.022627 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:14.022544    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:14.022627 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:14.022607    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:14.140274 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:14.140204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44hld" event={"ID":"93ce6edf-e0bf-4f03-9515-2d737ac88928","Type":"ContainerStarted","Data":"5dbd7375a2264862a809eca245243f66ea7d8a60c8b131dbf4ba45f81eba0d21"}
Sep 06 10:22:14.142376 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:14.142351    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54" event={"ID":"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a","Type":"ContainerStarted","Data":"f0b2bb03734cff1ee2d9b2c5ba5fffed843cf6dd88f7dc80c5d2068ec90ee044"}
Sep 06 10:22:14.145662 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:14.145644    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:22:14.146100 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:14.146061    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" event={"ID":"c7e1f1e1-44aa-4007-a88d-58ad1c176394","Type":"ContainerStarted","Data":"b1023deaed437bb1fc10eb763d3a8632eb146d5b8164ba365bd5306ca2160a20"}
Sep 06 10:22:14.146644 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:14.146594    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:22:14.146901 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:14.146881    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qk4m4"
Sep 06 10:22:14.159400 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:14.159352    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-44hld" podStartSLOduration=4.563041467 podStartE2EDuration="22.159334599s" totalImagesPullingTime="17.596293132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:14.15854202 +0000 UTC m=+22.642042148" watchObservedRunningTime="2026-09-06 10:22:14.159334599 +0000 UTC m=+22.642834711"
Sep 06 10:22:15.151141 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:15.151052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54" event={"ID":"f8c42e73-def7-412e-8dd2-4d9c9ad7ce8a","Type":"ContainerStarted","Data":"aebedf3b6dca2f863adfb4175998a2d5d4b4698e2509ca6ee5c40f64325d7083"}
Sep 06 10:22:15.169725 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:15.169683    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkd54" podStartSLOduration=3.540263712 podStartE2EDuration="23.169664534s" totalImagesPullingTime="19.629400822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:15.169290309 +0000 UTC m=+23.652790439" watchObservedRunningTime="2026-09-06 10:22:15.169664534 +0000 UTC m=+23.653164645"
Sep 06 10:22:16.022410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:16.022377    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:16.022683 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:16.022382    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:16.022683 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:16.022516    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:16.022683 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:16.022377    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:16.022683 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:16.022597    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:16.022882 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:16.022702    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:16.157571 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:16.157540    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:22:16.158150 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:16.158058    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" event={"ID":"c7e1f1e1-44aa-4007-a88d-58ad1c176394","Type":"ContainerStarted","Data":"45dc4058d44c5c91120793496fc449f2c0db88828716ed3eb4e0a173fd318ee4"}
Sep 06 10:22:18.023491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:18.023303    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:18.024242 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:18.023303    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:18.024242 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:18.023594    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:18.024242 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:18.023380    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:18.024242 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:18.023671    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:18.024242 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:18.023735    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:18.164481 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:18.164442    2564 generic.go:356] "Generic (PLEG): container finished" podID="7fa465d8-bbd3-4102-ae4f-7bba5685b27d" containerID="e906cae650daad2213326b1452a5c4fad7c81e5a6d75b835ab3fb779dda8e1a7" exitCode=0
Sep 06 10:22:18.164639 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:18.164484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-668b9" event={"ID":"7fa465d8-bbd3-4102-ae4f-7bba5685b27d","Type":"ContainerDied","Data":"e906cae650daad2213326b1452a5c4fad7c81e5a6d75b835ab3fb779dda8e1a7"}
Sep 06 10:22:18.168076 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:18.168059    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:22:18.168453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:18.168437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" event={"ID":"c7e1f1e1-44aa-4007-a88d-58ad1c176394","Type":"ContainerStarted","Data":"f4a5d69bdcd5c9290e4fd78f42c383f1e6bc7b17aa495ff80ca1fa9a644c4839"}
Sep 06 10:22:18.168758 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:18.168735    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:22:18.168953 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:18.168939    2564 scope.go:117] "RemoveContainer" containerID="b40b37ce831e80349a9bf00e60055196e3590eb25f795e8e91c329a58111f94a"
Sep 06 10:22:18.184826 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:18.184801    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:22:19.172895 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.172574    2564 generic.go:356] "Generic (PLEG): container finished" podID="7fa465d8-bbd3-4102-ae4f-7bba5685b27d" containerID="96fabf3f22893a9b8b0b1b27e60cd80866a4b48f24b244671ff91fd693561afc" exitCode=0
Sep 06 10:22:19.172895 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.172665    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-668b9" event={"ID":"7fa465d8-bbd3-4102-ae4f-7bba5685b27d","Type":"ContainerDied","Data":"96fabf3f22893a9b8b0b1b27e60cd80866a4b48f24b244671ff91fd693561afc"}
Sep 06 10:22:19.176880 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.176863    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:22:19.177264 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.177237    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" event={"ID":"c7e1f1e1-44aa-4007-a88d-58ad1c176394","Type":"ContainerStarted","Data":"0ad4e7b0b35fdd03bfeab82914bd5957efa03412ed10927dc65e82676a293f40"}
Sep 06 10:22:19.177354 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.177333    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 06 10:22:19.177516 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.177497    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:22:19.192804 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.192781    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:22:19.218276 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.218239    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh" podStartSLOduration=9.590031942 podStartE2EDuration="27.218224876s" totalImagesPullingTime="17.628192934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:19.216839222 +0000 UTC m=+27.700339332" watchObservedRunningTime="2026-09-06 10:22:19.218224876 +0000 UTC m=+27.701724976"
Sep 06 10:22:19.268403 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.268377    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hpfqd"]
Sep 06 10:22:19.268566 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.268488    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:19.268614 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:19.268580    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:19.271563 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.271541    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2fg8"]
Sep 06 10:22:19.271704 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.271639    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:19.271759 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:19.271722    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:19.272160 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.272082    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w7hwz"]
Sep 06 10:22:19.272267 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:19.272228    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:19.272358 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:19.272336    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:20.081042 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:20.081017    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:22:20.181528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:20.181443    2564 generic.go:356] "Generic (PLEG): container finished" podID="7fa465d8-bbd3-4102-ae4f-7bba5685b27d" containerID="ebfff186b7c5f8e91ac0cb4fbd28a7b26a00f3484d604d11506f1cae96ecc5c5" exitCode=0
Sep 06 10:22:20.181874 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:20.181527    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-668b9" event={"ID":"7fa465d8-bbd3-4102-ae4f-7bba5685b27d","Type":"ContainerDied","Data":"ebfff186b7c5f8e91ac0cb4fbd28a7b26a00f3484d604d11506f1cae96ecc5c5"}
Sep 06 10:22:21.023296 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:21.023195    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:21.023296 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:21.023253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:21.023296 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:21.023252    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:21.023528 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:21.023345    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:21.023528 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:21.023432    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:21.023528 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:21.023482    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:23.022930 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:23.022895    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:23.023470 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:23.022891    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:23.023470 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:23.023045    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:23.023470 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:23.022905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:23.023470 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:23.023134    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:23.023470 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:23.023175    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:25.023271 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.023011    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:25.023271 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.023070    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:25.023773 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.023078    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:25.023773 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.023314    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hpfqd" podUID="6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d"
Sep 06 10:22:25.023773 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.023401    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2fg8" podUID="cc37245a-69d3-4f71-9011-40a62c5457a2"
Sep 06 10:22:25.023773 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.023474    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w7hwz" podUID="21ec7fd0-b053-4c24-b252-a32a59409d59"
Sep 06 10:22:25.196460 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.196428    2564 generic.go:356] "Generic (PLEG): container finished" podID="7fa465d8-bbd3-4102-ae4f-7bba5685b27d" containerID="a078d91bfc8942bc279c4882df51d45d5bae5f09aa1a0036ac23ac7dd8735e79" exitCode=0
Sep 06 10:22:25.196597 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.196473    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-668b9" event={"ID":"7fa465d8-bbd3-4102-ae4f-7bba5685b27d","Type":"ContainerDied","Data":"a078d91bfc8942bc279c4882df51d45d5bae5f09aa1a0036ac23ac7dd8735e79"}
Sep 06 10:22:25.653485 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.653394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47\") pod \"network-check-target-m2fg8\" (UID: \"cc37245a-69d3-4f71-9011-40a62c5457a2\") " pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:25.653733 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.653524    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:22:25.653733 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.653538    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:22:25.653733 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.653548    2564 projected.go:196] Error preparing data for projected volume kube-api-access-bst47 for pod openshift-network-diagnostics/network-check-target-m2fg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:22:25.653733 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.653599    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47 podName:cc37245a-69d3-4f71-9011-40a62c5457a2 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:57.653585342 +0000 UTC m=+66.137085429 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bst47" (UniqueName: "kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47") pod "network-check-target-m2fg8" (UID: "cc37245a-69d3-4f71-9011-40a62c5457a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:22:25.754161 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.754123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:25.754340 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.754174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:25.754340 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.754295    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:22:25.754410 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.754356    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret podName:6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d nodeName:}" failed. No retries permitted until 2026-09-06 10:22:57.754340606 +0000 UTC m=+66.237840693 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret") pod "global-pull-secret-syncer-hpfqd" (UID: "6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:22:25.754410 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.754295    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:22:25.754485 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:25.754424    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs podName:21ec7fd0-b053-4c24-b252-a32a59409d59 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:57.754412159 +0000 UTC m=+66.237912261 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs") pod "network-metrics-daemon-w7hwz" (UID: "21ec7fd0-b053-4c24-b252-a32a59409d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:22:25.768338 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.768316    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeReady"
Sep 06 10:22:25.768476 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.768436    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 06 10:22:25.824106 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.824079    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7fpmw"]
Sep 06 10:22:25.845129 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.845106    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cbr79"]
Sep 06 10:22:25.845271 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.845239    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7fpmw"
Sep 06 10:22:25.847949 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.847930    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 06 10:22:25.848084 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.847994    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 06 10:22:25.848084 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.848018    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 06 10:22:25.848416 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.848402    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5f7rv\""
Sep 06 10:22:25.864601 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.864573    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7fpmw"]
Sep 06 10:22:25.864714 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.864609    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cbr79"]
Sep 06 10:22:25.864714 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.864690    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:25.867264 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.867244    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 06 10:22:25.867383 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.867246    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 06 10:22:25.867383 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.867294    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mhwb9\""
Sep 06 10:22:25.955242 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.955144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d63a459-955c-424e-ad42-e5d112a2e6d9-cert\") pod \"ingress-canary-7fpmw\" (UID: \"7d63a459-955c-424e-ad42-e5d112a2e6d9\") " pod="openshift-ingress-canary/ingress-canary-7fpmw"
Sep 06 10:22:25.955242 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.955176    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44xh\" (UniqueName: \"kubernetes.io/projected/7d63a459-955c-424e-ad42-e5d112a2e6d9-kube-api-access-z44xh\") pod \"ingress-canary-7fpmw\" (UID: \"7d63a459-955c-424e-ad42-e5d112a2e6d9\") " pod="openshift-ingress-canary/ingress-canary-7fpmw"
Sep 06 10:22:25.955242 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.955237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:25.955442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.955275    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7h7k\" (UniqueName: \"kubernetes.io/projected/3e262594-dced-40d6-b678-a0d04be6252b-kube-api-access-d7h7k\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:25.955442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.955346    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e262594-dced-40d6-b678-a0d04be6252b-tmp-dir\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:25.955442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:25.955367    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e262594-dced-40d6-b678-a0d04be6252b-config-volume\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:26.055837 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.055805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7h7k\" (UniqueName: \"kubernetes.io/projected/3e262594-dced-40d6-b678-a0d04be6252b-kube-api-access-d7h7k\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:26.056347 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.055873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e262594-dced-40d6-b678-a0d04be6252b-tmp-dir\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:26.056347 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.055900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e262594-dced-40d6-b678-a0d04be6252b-config-volume\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:26.056347 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.055918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d63a459-955c-424e-ad42-e5d112a2e6d9-cert\") pod \"ingress-canary-7fpmw\" (UID: \"7d63a459-955c-424e-ad42-e5d112a2e6d9\") " pod="openshift-ingress-canary/ingress-canary-7fpmw"
Sep 06 10:22:26.056347 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.055935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44xh\" (UniqueName: \"kubernetes.io/projected/7d63a459-955c-424e-ad42-e5d112a2e6d9-kube-api-access-z44xh\") pod \"ingress-canary-7fpmw\" (UID: \"7d63a459-955c-424e-ad42-e5d112a2e6d9\") " pod="openshift-ingress-canary/ingress-canary-7fpmw"
Sep 06 10:22:26.056347 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.056003    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:26.056347 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:26.056109    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:26.056347 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:26.056166    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls podName:3e262594-dced-40d6-b678-a0d04be6252b nodeName:}" failed. No retries permitted until 2026-09-06 10:22:26.556148693 +0000 UTC m=+35.039648783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls") pod "dns-default-cbr79" (UID: "3e262594-dced-40d6-b678-a0d04be6252b") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:26.056607 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.056385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e262594-dced-40d6-b678-a0d04be6252b-tmp-dir\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:26.056607 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.056505    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e262594-dced-40d6-b678-a0d04be6252b-config-volume\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:26.059931 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.059911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d63a459-955c-424e-ad42-e5d112a2e6d9-cert\") pod \"ingress-canary-7fpmw\" (UID: \"7d63a459-955c-424e-ad42-e5d112a2e6d9\") " pod="openshift-ingress-canary/ingress-canary-7fpmw"
Sep 06 10:22:26.064463 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.064443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7h7k\" (UniqueName: \"kubernetes.io/projected/3e262594-dced-40d6-b678-a0d04be6252b-kube-api-access-d7h7k\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:26.064572 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.064531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44xh\" (UniqueName: \"kubernetes.io/projected/7d63a459-955c-424e-ad42-e5d112a2e6d9-kube-api-access-z44xh\") pod \"ingress-canary-7fpmw\" (UID: \"7d63a459-955c-424e-ad42-e5d112a2e6d9\") " pod="openshift-ingress-canary/ingress-canary-7fpmw"
Sep 06 10:22:26.154488 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.154458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7fpmw"
Sep 06 10:22:26.204624 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.204595    2564 generic.go:356] "Generic (PLEG): container finished" podID="7fa465d8-bbd3-4102-ae4f-7bba5685b27d" containerID="c7f130e69bf9b75a2731d5efa376c996f764778a4d45a44a7e3c92d11ad01082" exitCode=0
Sep 06 10:22:26.204741 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.204647    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-668b9" event={"ID":"7fa465d8-bbd3-4102-ae4f-7bba5685b27d","Type":"ContainerDied","Data":"c7f130e69bf9b75a2731d5efa376c996f764778a4d45a44a7e3c92d11ad01082"}
Sep 06 10:22:26.487667 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.487415    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7fpmw"]
Sep 06 10:22:26.490969 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:22:26.490940    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d63a459_955c_424e_ad42_e5d112a2e6d9.slice/crio-2b53f9ef647b7fdf1e7187cce233606860635b0c23c803d1e5e1a36571431641 WatchSource:0}: Error finding container 2b53f9ef647b7fdf1e7187cce233606860635b0c23c803d1e5e1a36571431641: Status 404 returned error can't find the container with id 2b53f9ef647b7fdf1e7187cce233606860635b0c23c803d1e5e1a36571431641
Sep 06 10:22:26.560966 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:26.560936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:26.561108 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:26.561037    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:26.561108 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:26.561092    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls podName:3e262594-dced-40d6-b678-a0d04be6252b nodeName:}" failed. No retries permitted until 2026-09-06 10:22:27.561076414 +0000 UTC m=+36.044576501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls") pod "dns-default-cbr79" (UID: "3e262594-dced-40d6-b678-a0d04be6252b") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:27.022875 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.022842    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:27.023111 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.022955    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:27.023111 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.023048    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:27.026550 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.026500    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 10:22:27.026668 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.026576    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 10:22:27.026668 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.026649    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 10:22:27.026774 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.026691    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 10:22:27.026774 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.026725    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xrhk7\""
Sep 06 10:22:27.026873 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.026790    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xmc8p\""
Sep 06 10:22:27.211940 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.211904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-668b9" event={"ID":"7fa465d8-bbd3-4102-ae4f-7bba5685b27d","Type":"ContainerStarted","Data":"889d5f7c7ad67aa58e95c48adc2fe390c488c491230276385aa0da1ab3928fa5"}
Sep 06 10:22:27.213082 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.213061    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7fpmw" event={"ID":"7d63a459-955c-424e-ad42-e5d112a2e6d9","Type":"ContainerStarted","Data":"2b53f9ef647b7fdf1e7187cce233606860635b0c23c803d1e5e1a36571431641"}
Sep 06 10:22:27.233526 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.233474    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-668b9" podStartSLOduration=7.495501702 podStartE2EDuration="35.233460105s" totalImagesPullingTime="27.737958403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:27.232157294 +0000 UTC m=+35.715657444" watchObservedRunningTime="2026-09-06 10:22:27.233460105 +0000 UTC m=+35.716960213"
Sep 06 10:22:27.569193 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:27.569158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:27.569380 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:27.569323    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:27.569434 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:27.569404    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls podName:3e262594-dced-40d6-b678-a0d04be6252b nodeName:}" failed. No retries permitted until 2026-09-06 10:22:29.569386148 +0000 UTC m=+38.052886239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls") pod "dns-default-cbr79" (UID: "3e262594-dced-40d6-b678-a0d04be6252b") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:29.220066 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:29.219985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7fpmw" event={"ID":"7d63a459-955c-424e-ad42-e5d112a2e6d9","Type":"ContainerStarted","Data":"bf6f3bcfe720af800129dc5ea9a0829048b071d1e5176bd5dbb4878c75a53c49"}
Sep 06 10:22:29.237762 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:29.237716    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7fpmw" podStartSLOduration=1.811954338 podStartE2EDuration="4.237699983s" totalImagesPullingTime="2.425745645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:29.237251699 +0000 UTC m=+37.720751829" watchObservedRunningTime="2026-09-06 10:22:29.237699983 +0000 UTC m=+37.721200118"
Sep 06 10:22:29.585474 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:29.585447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:29.585624 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:29.585605    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:29.585683 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:29.585673    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls podName:3e262594-dced-40d6-b678-a0d04be6252b nodeName:}" failed. No retries permitted until 2026-09-06 10:22:33.58565422 +0000 UTC m=+42.069154528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls") pod "dns-default-cbr79" (UID: "3e262594-dced-40d6-b678-a0d04be6252b") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:33.615106 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.614893    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:33.615106 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:33.615069    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:33.615546 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:33.615161    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls podName:3e262594-dced-40d6-b678-a0d04be6252b nodeName:}" failed. No retries permitted until 2026-09-06 10:22:41.615145129 +0000 UTC m=+50.098645216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls") pod "dns-default-cbr79" (UID: "3e262594-dced-40d6-b678-a0d04be6252b") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:33.663667 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.663630    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-25jrg"]
Sep 06 10:22:33.698577 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.698546    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-25jrg"]
Sep 06 10:22:33.698715 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.698669    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.701178 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.701145    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 06 10:22:33.701369 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.701349    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p462p\""
Sep 06 10:22:33.701450 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.701386    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 06 10:22:33.701450 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.701406    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 06 10:22:33.702236 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.702202    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 06 10:22:33.819466 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.817094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c23f4e2-c759-4b6a-af25-62e437473b3a-crio-socket\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.819466 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.817165    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c23f4e2-c759-4b6a-af25-62e437473b3a-data-volume\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.819466 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.817200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.819466 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.817319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.819466 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.817400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm24w\" (UniqueName: \"kubernetes.io/projected/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-api-access-gm24w\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.917879 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.917795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.917879 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.917860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm24w\" (UniqueName: \"kubernetes.io/projected/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-api-access-gm24w\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.917879 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.917879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c23f4e2-c759-4b6a-af25-62e437473b3a-crio-socket\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.918082 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.917904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c23f4e2-c759-4b6a-af25-62e437473b3a-data-volume\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.918082 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.917918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.918082 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:33.917964    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:33.918082 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:33.918021    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 10:22:33.918082 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:33.918040    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls podName:4c23f4e2-c759-4b6a-af25-62e437473b3a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:34.418023056 +0000 UTC m=+42.901523146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-25jrg" (UID: "4c23f4e2-c759-4b6a-af25-62e437473b3a") : secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:33.918082 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:33.918080    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm podName:4c23f4e2-c759-4b6a-af25-62e437473b3a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:34.418065666 +0000 UTC m=+42.901565752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-25jrg" (UID: "4c23f4e2-c759-4b6a-af25-62e437473b3a") : configmap "kube-rbac-proxy" not found
Sep 06 10:22:33.918325 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.918112    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c23f4e2-c759-4b6a-af25-62e437473b3a-crio-socket\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.918325 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.918266    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c23f4e2-c759-4b6a-af25-62e437473b3a-data-volume\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:33.927820 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:33.927786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm24w\" (UniqueName: \"kubernetes.io/projected/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-api-access-gm24w\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:34.421165 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:34.421123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:34.421348 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:34.421206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:34.421348 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:34.421273    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 10:22:34.421421 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:34.421353    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm podName:4c23f4e2-c759-4b6a-af25-62e437473b3a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:35.421336779 +0000 UTC m=+43.904836885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-25jrg" (UID: "4c23f4e2-c759-4b6a-af25-62e437473b3a") : configmap "kube-rbac-proxy" not found
Sep 06 10:22:34.421421 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:34.421352    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:34.421421 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:34.421381    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls podName:4c23f4e2-c759-4b6a-af25-62e437473b3a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:35.421375668 +0000 UTC m=+43.904875754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-25jrg" (UID: "4c23f4e2-c759-4b6a-af25-62e437473b3a") : secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:35.322566 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.322531    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h"]
Sep 06 10:22:35.354165 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.354132    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h"]
Sep 06 10:22:35.354332 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.354262    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h"
Sep 06 10:22:35.356969 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.356941    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 06 10:22:35.356969 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.356929    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 06 10:22:35.357137 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.357062    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gp7nk\""
Sep 06 10:22:35.429502 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.429468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:35.429666 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.429539    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:35.429666 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:35.429588    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 10:22:35.429666 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:35.429641    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:35.429774 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:35.429667    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm podName:4c23f4e2-c759-4b6a-af25-62e437473b3a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:37.429645639 +0000 UTC m=+45.913145731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-25jrg" (UID: "4c23f4e2-c759-4b6a-af25-62e437473b3a") : configmap "kube-rbac-proxy" not found
Sep 06 10:22:35.429774 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:35.429684    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls podName:4c23f4e2-c759-4b6a-af25-62e437473b3a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:37.429677161 +0000 UTC m=+45.913177248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-25jrg" (UID: "4c23f4e2-c759-4b6a-af25-62e437473b3a") : secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:35.530657 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.530615    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6psw8\" (UniqueName: \"kubernetes.io/projected/31d7c255-86b6-45b2-a235-577e6b069272-kube-api-access-6psw8\") pod \"migrator-fdbb8fdff-86x2h\" (UID: \"31d7c255-86b6-45b2-a235-577e6b069272\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h"
Sep 06 10:22:35.617708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.617632    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mjk44_7e2d842b-52a5-4065-a84b-1e8b047e2ba7/dns-node-resolver/0.log"
Sep 06 10:22:35.631914 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.631887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6psw8\" (UniqueName: \"kubernetes.io/projected/31d7c255-86b6-45b2-a235-577e6b069272-kube-api-access-6psw8\") pod \"migrator-fdbb8fdff-86x2h\" (UID: \"31d7c255-86b6-45b2-a235-577e6b069272\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h"
Sep 06 10:22:35.640095 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.640073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6psw8\" (UniqueName: \"kubernetes.io/projected/31d7c255-86b6-45b2-a235-577e6b069272-kube-api-access-6psw8\") pod \"migrator-fdbb8fdff-86x2h\" (UID: \"31d7c255-86b6-45b2-a235-577e6b069272\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h"
Sep 06 10:22:35.663011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.662982    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h"
Sep 06 10:22:35.911269 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.911160    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-dhkn2"]
Sep 06 10:22:35.935550 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.935518    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-dhkn2"]
Sep 06 10:22:35.935707 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.935624    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:35.938407 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.938383    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 06 10:22:35.938544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.938434    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 06 10:22:35.938544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.938476    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 06 10:22:35.939467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.939451    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-99wcx\""
Sep 06 10:22:35.939467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.939459    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 06 10:22:35.962448 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:35.962402    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h"]
Sep 06 10:22:35.965606 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:22:35.965581    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d7c255_86b6_45b2_a235_577e6b069272.slice/crio-b43cdc9fbac7a1079c3cc87b30d2720ecd340329e6c684a0f066e7eb3984e20f WatchSource:0}: Error finding container b43cdc9fbac7a1079c3cc87b30d2720ecd340329e6c684a0f066e7eb3984e20f: Status 404 returned error can't find the container with id b43cdc9fbac7a1079c3cc87b30d2720ecd340329e6c684a0f066e7eb3984e20f
Sep 06 10:22:36.034925 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.034894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/34191bf2-5a84-4082-aef3-e1f574fe0b50-signing-cabundle\") pod \"service-ca-75d99c484-dhkn2\" (UID: \"34191bf2-5a84-4082-aef3-e1f574fe0b50\") " pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:36.035103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.034954    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/34191bf2-5a84-4082-aef3-e1f574fe0b50-signing-key\") pod \"service-ca-75d99c484-dhkn2\" (UID: \"34191bf2-5a84-4082-aef3-e1f574fe0b50\") " pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:36.035103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.035002    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmv6b\" (UniqueName: \"kubernetes.io/projected/34191bf2-5a84-4082-aef3-e1f574fe0b50-kube-api-access-nmv6b\") pod \"service-ca-75d99c484-dhkn2\" (UID: \"34191bf2-5a84-4082-aef3-e1f574fe0b50\") " pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:36.135735 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.135700    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/34191bf2-5a84-4082-aef3-e1f574fe0b50-signing-key\") pod \"service-ca-75d99c484-dhkn2\" (UID: \"34191bf2-5a84-4082-aef3-e1f574fe0b50\") " pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:36.135735 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.135734    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmv6b\" (UniqueName: \"kubernetes.io/projected/34191bf2-5a84-4082-aef3-e1f574fe0b50-kube-api-access-nmv6b\") pod \"service-ca-75d99c484-dhkn2\" (UID: \"34191bf2-5a84-4082-aef3-e1f574fe0b50\") " pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:36.135927 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.135799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/34191bf2-5a84-4082-aef3-e1f574fe0b50-signing-cabundle\") pod \"service-ca-75d99c484-dhkn2\" (UID: \"34191bf2-5a84-4082-aef3-e1f574fe0b50\") " pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:36.136569 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.136550    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/34191bf2-5a84-4082-aef3-e1f574fe0b50-signing-cabundle\") pod \"service-ca-75d99c484-dhkn2\" (UID: \"34191bf2-5a84-4082-aef3-e1f574fe0b50\") " pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:36.138014 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.137994    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/34191bf2-5a84-4082-aef3-e1f574fe0b50-signing-key\") pod \"service-ca-75d99c484-dhkn2\" (UID: \"34191bf2-5a84-4082-aef3-e1f574fe0b50\") " pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:36.143583 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.143559    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmv6b\" (UniqueName: \"kubernetes.io/projected/34191bf2-5a84-4082-aef3-e1f574fe0b50-kube-api-access-nmv6b\") pod \"service-ca-75d99c484-dhkn2\" (UID: \"34191bf2-5a84-4082-aef3-e1f574fe0b50\") " pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:36.237508 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.237422    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h" event={"ID":"31d7c255-86b6-45b2-a235-577e6b069272","Type":"ContainerStarted","Data":"b43cdc9fbac7a1079c3cc87b30d2720ecd340329e6c684a0f066e7eb3984e20f"}
Sep 06 10:22:36.244722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.244702    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-dhkn2"
Sep 06 10:22:36.417095 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.417070    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5wxtv_7ad43d1d-8428-4eae-a728-b9052f618103/node-ca/0.log"
Sep 06 10:22:36.544587 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:36.544556    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-dhkn2"]
Sep 06 10:22:36.548897 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:22:36.548870    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34191bf2_5a84_4082_aef3_e1f574fe0b50.slice/crio-3f4efdd9764f695fcc34b8d39b4577fbd6a5f07e733b3bddea8806b30986c03e WatchSource:0}: Error finding container 3f4efdd9764f695fcc34b8d39b4577fbd6a5f07e733b3bddea8806b30986c03e: Status 404 returned error can't find the container with id 3f4efdd9764f695fcc34b8d39b4577fbd6a5f07e733b3bddea8806b30986c03e
Sep 06 10:22:37.217865 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:37.217832    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7fpmw_7d63a459-955c-424e-ad42-e5d112a2e6d9/serve-healthcheck-canary/0.log"
Sep 06 10:22:37.241755 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:37.241714    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-dhkn2" event={"ID":"34191bf2-5a84-4082-aef3-e1f574fe0b50","Type":"ContainerStarted","Data":"3f4efdd9764f695fcc34b8d39b4577fbd6a5f07e733b3bddea8806b30986c03e"}
Sep 06 10:22:37.445984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:37.445941    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:37.446445 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:37.446113    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:37.446445 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:37.446132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:37.446445 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:37.446226    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 10:22:37.446445 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:37.446237    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls podName:4c23f4e2-c759-4b6a-af25-62e437473b3a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:41.446203667 +0000 UTC m=+49.929703754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-25jrg" (UID: "4c23f4e2-c759-4b6a-af25-62e437473b3a") : secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:37.446445 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:37.446278    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm podName:4c23f4e2-c759-4b6a-af25-62e437473b3a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:41.446261978 +0000 UTC m=+49.929762082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-25jrg" (UID: "4c23f4e2-c759-4b6a-af25-62e437473b3a") : configmap "kube-rbac-proxy" not found
Sep 06 10:22:38.247259 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:38.247153    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h" event={"ID":"31d7c255-86b6-45b2-a235-577e6b069272","Type":"ContainerStarted","Data":"3bae48ebd410c06ebf2f2227cbf1fc52352f57bd94515ec05bff353c1516aead"}
Sep 06 10:22:38.247259 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:38.247239    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h" event={"ID":"31d7c255-86b6-45b2-a235-577e6b069272","Type":"ContainerStarted","Data":"55e6719a7bc610c960342bb754d1daac360773d5d30c311dfbac7172866f80ef"}
Sep 06 10:22:38.262696 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:38.262633    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-86x2h" podStartSLOduration=1.614474173 podStartE2EDuration="3.262617973s" totalImagesPullingTime="1.6481438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:38.261613341 +0000 UTC m=+46.745113451" watchObservedRunningTime="2026-09-06 10:22:38.262617973 +0000 UTC m=+46.746118081"
Sep 06 10:22:39.251565 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:39.251524    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-dhkn2" event={"ID":"34191bf2-5a84-4082-aef3-e1f574fe0b50","Type":"ContainerStarted","Data":"a69a3deab3abc06babd8d76054578753fa745ff2ed2f8dbd6fcc457a4ac3d8e4"}
Sep 06 10:22:39.266791 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:39.266740    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-dhkn2" podStartSLOduration=2.032234242 podStartE2EDuration="4.266724395s" totalImagesPullingTime="2.234490153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:39.266057464 +0000 UTC m=+47.749557591" watchObservedRunningTime="2026-09-06 10:22:39.266724395 +0000 UTC m=+47.750224553"
Sep 06 10:22:41.477540 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:41.477496    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:41.477948 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:41.477564    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:41.477948 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:41.477663    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:41.477948 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:41.477731    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls podName:4c23f4e2-c759-4b6a-af25-62e437473b3a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:49.477716306 +0000 UTC m=+57.961216392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-25jrg" (UID: "4c23f4e2-c759-4b6a-af25-62e437473b3a") : secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:41.478059 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:41.477990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23f4e2-c759-4b6a-af25-62e437473b3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:41.678240 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:41.678189    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:41.678453 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:41.678310    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:41.678453 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:22:41.678363    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls podName:3e262594-dced-40d6-b678-a0d04be6252b nodeName:}" failed. No retries permitted until 2026-09-06 10:22:57.678349568 +0000 UTC m=+66.161849667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls") pod "dns-default-cbr79" (UID: "3e262594-dced-40d6-b678-a0d04be6252b") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:49.540035 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:49.539978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:49.542319 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:49.542298    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23f4e2-c759-4b6a-af25-62e437473b3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25jrg\" (UID: \"4c23f4e2-c759-4b6a-af25-62e437473b3a\") " pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:49.607761 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:49.607718    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-25jrg"
Sep 06 10:22:49.878621 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:49.878540    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-25jrg"]
Sep 06 10:22:49.881410 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:22:49.881379    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c23f4e2_c759_4b6a_af25_62e437473b3a.slice/crio-d6c0a1a656bb596f52505dbaa42920b20fdac7298df7e23137671530c27d2d2c WatchSource:0}: Error finding container d6c0a1a656bb596f52505dbaa42920b20fdac7298df7e23137671530c27d2d2c: Status 404 returned error can't find the container with id d6c0a1a656bb596f52505dbaa42920b20fdac7298df7e23137671530c27d2d2c
Sep 06 10:22:50.281690 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:50.281599    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-25jrg" event={"ID":"4c23f4e2-c759-4b6a-af25-62e437473b3a","Type":"ContainerStarted","Data":"96cc1d27881d4fef5c7e74f2fb3d49ca03af353994865e52ac8a8787bdeda94d"}
Sep 06 10:22:50.281690 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:50.281662    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-25jrg" event={"ID":"4c23f4e2-c759-4b6a-af25-62e437473b3a","Type":"ContainerStarted","Data":"d6c0a1a656bb596f52505dbaa42920b20fdac7298df7e23137671530c27d2d2c"}
Sep 06 10:22:51.195705 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:51.195676    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bsdqh"
Sep 06 10:22:51.287863 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:51.287823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-25jrg" event={"ID":"4c23f4e2-c759-4b6a-af25-62e437473b3a","Type":"ContainerStarted","Data":"675d61cf0837f3a86b31c725fce6b31b51b502449d6a2ac345d88dcc2f48783d"}
Sep 06 10:22:52.293097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:52.293060    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-25jrg" event={"ID":"4c23f4e2-c759-4b6a-af25-62e437473b3a","Type":"ContainerStarted","Data":"9bfcf3a19c881706c6ab534fc089599482e017bf0d99450163fdba4e4d7a97d5"}
Sep 06 10:22:52.311447 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:52.311397    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-25jrg" podStartSLOduration=17.295551157 podStartE2EDuration="19.311378613s" totalImagesPullingTime="2.015827456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:22:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:52.311264965 +0000 UTC m=+60.794765085" watchObservedRunningTime="2026-09-06 10:22:52.311378613 +0000 UTC m=+60.794878721"
Sep 06 10:22:55.611770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.611734    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"]
Sep 06 10:22:55.632440 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.632408    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"]
Sep 06 10:22:55.632599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.632502    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"
Sep 06 10:22:55.636249 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.636190    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 06 10:22:55.636394 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.636239    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dbq5q\""
Sep 06 10:22:55.636394 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.636190    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 06 10:22:55.636394 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.636239    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 06 10:22:55.636394 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.636190    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 06 10:22:55.688587 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.688550    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57a1b94c-1988-43d2-a4c8-447ed0464d16-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cf6944c96-wtgr4\" (UID: \"57a1b94c-1988-43d2-a4c8-447ed0464d16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"
Sep 06 10:22:55.688754 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.688608    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mjk8\" (UniqueName: \"kubernetes.io/projected/57a1b94c-1988-43d2-a4c8-447ed0464d16-kube-api-access-6mjk8\") pod \"managed-serviceaccount-addon-agent-7cf6944c96-wtgr4\" (UID: \"57a1b94c-1988-43d2-a4c8-447ed0464d16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"
Sep 06 10:22:55.747286 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.747254    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f64dddb67-t4fz6"]
Sep 06 10:22:55.768196 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.768172    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.770919 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.770893    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 06 10:22:55.771074 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.770972    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 06 10:22:55.771243 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.771227    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4td26\""
Sep 06 10:22:55.771309 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.771284    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 06 10:22:55.775547 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.775526    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f64dddb67-t4fz6"]
Sep 06 10:22:55.776400 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.776377    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 06 10:22:55.789155 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.789136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57a1b94c-1988-43d2-a4c8-447ed0464d16-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cf6944c96-wtgr4\" (UID: \"57a1b94c-1988-43d2-a4c8-447ed0464d16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"
Sep 06 10:22:55.789273 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.789176    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjk8\" (UniqueName: \"kubernetes.io/projected/57a1b94c-1988-43d2-a4c8-447ed0464d16-kube-api-access-6mjk8\") pod \"managed-serviceaccount-addon-agent-7cf6944c96-wtgr4\" (UID: \"57a1b94c-1988-43d2-a4c8-447ed0464d16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"
Sep 06 10:22:55.791524 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.791503    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57a1b94c-1988-43d2-a4c8-447ed0464d16-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cf6944c96-wtgr4\" (UID: \"57a1b94c-1988-43d2-a4c8-447ed0464d16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"
Sep 06 10:22:55.803904 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.803881    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjk8\" (UniqueName: \"kubernetes.io/projected/57a1b94c-1988-43d2-a4c8-447ed0464d16-kube-api-access-6mjk8\") pod \"managed-serviceaccount-addon-agent-7cf6944c96-wtgr4\" (UID: \"57a1b94c-1988-43d2-a4c8-447ed0464d16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"
Sep 06 10:22:55.889677 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.889598    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-registry-tls\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.889677 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.889634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-bound-sa-token\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.889677 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.889658    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4h5\" (UniqueName: \"kubernetes.io/projected/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-kube-api-access-qj4h5\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.889953 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.889711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-image-registry-private-configuration\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.889953 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.889744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-installation-pull-secrets\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.889953 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.889811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-trusted-ca\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.889953 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.889896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-ca-trust-extracted\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.890109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.889956    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-registry-certificates\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.952949 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.952920    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"
Sep 06 10:22:55.990916 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.990883    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-ca-trust-extracted\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.991032 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.990948    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-registry-certificates\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.991032 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.990984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-registry-tls\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.991032 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.990999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-bound-sa-token\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.991032 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.991017    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4h5\" (UniqueName: \"kubernetes.io/projected/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-kube-api-access-qj4h5\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.991290 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.991049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-image-registry-private-configuration\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.991290 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.991087    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-installation-pull-secrets\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.991290 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.991131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-trusted-ca\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.991456 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.991326    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-ca-trust-extracted\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.991933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.991885    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-registry-certificates\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.992231 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.992200    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-trusted-ca\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.993564 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.993537    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-registry-tls\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.993645 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.993613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-image-registry-private-configuration\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:55.993885 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:55.993865    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-installation-pull-secrets\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:56.000806 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:56.000785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-bound-sa-token\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:56.001067 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:56.001048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4h5\" (UniqueName: \"kubernetes.io/projected/71bd4b2d-5317-4a86-9bbf-9995f3b667f2-kube-api-access-qj4h5\") pod \"image-registry-6f64dddb67-t4fz6\" (UID: \"71bd4b2d-5317-4a86-9bbf-9995f3b667f2\") " pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:56.078447 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:56.078416    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:56.243621 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:56.243590    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4"]
Sep 06 10:22:56.246443 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:22:56.246418    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57a1b94c_1988_43d2_a4c8_447ed0464d16.slice/crio-e002d64f6bc9e65ef25f03a9fa7f3fb4ee0f1ace0054d3c8431682af518b6b48 WatchSource:0}: Error finding container e002d64f6bc9e65ef25f03a9fa7f3fb4ee0f1ace0054d3c8431682af518b6b48: Status 404 returned error can't find the container with id e002d64f6bc9e65ef25f03a9fa7f3fb4ee0f1ace0054d3c8431682af518b6b48
Sep 06 10:22:56.309327 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:56.309287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4" event={"ID":"57a1b94c-1988-43d2-a4c8-447ed0464d16","Type":"ContainerStarted","Data":"e002d64f6bc9e65ef25f03a9fa7f3fb4ee0f1ace0054d3c8431682af518b6b48"}
Sep 06 10:22:56.381462 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:56.381427    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f64dddb67-t4fz6"]
Sep 06 10:22:56.396293 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:22:56.396264    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bd4b2d_5317_4a86_9bbf_9995f3b667f2.slice/crio-5db70bd0db4d1173a9fe1da8123b3fcdb0e40ce5f72e2ecb6d8a9f5e7646c621 WatchSource:0}: Error finding container 5db70bd0db4d1173a9fe1da8123b3fcdb0e40ce5f72e2ecb6d8a9f5e7646c621: Status 404 returned error can't find the container with id 5db70bd0db4d1173a9fe1da8123b3fcdb0e40ce5f72e2ecb6d8a9f5e7646c621
Sep 06 10:22:57.315463 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.315427    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6" event={"ID":"71bd4b2d-5317-4a86-9bbf-9995f3b667f2","Type":"ContainerStarted","Data":"3800b69eb629d308dc6b430c789dc77a347178d38a31e2a16077c5d50b47e749"}
Sep 06 10:22:57.315892 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.315484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6" event={"ID":"71bd4b2d-5317-4a86-9bbf-9995f3b667f2","Type":"ContainerStarted","Data":"5db70bd0db4d1173a9fe1da8123b3fcdb0e40ce5f72e2ecb6d8a9f5e7646c621"}
Sep 06 10:22:57.315892 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.315604    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:22:57.336689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.336473    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6" podStartSLOduration=2.336453815 podStartE2EDuration="2.336453815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:57.335068123 +0000 UTC m=+65.818568236" watchObservedRunningTime="2026-09-06 10:22:57.336453815 +0000 UTC m=+65.819953925"
Sep 06 10:22:57.708703 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.708610    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47\") pod \"network-check-target-m2fg8\" (UID: \"cc37245a-69d3-4f71-9011-40a62c5457a2\") " pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:57.708961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.708707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:57.711201 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.711179    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e262594-dced-40d6-b678-a0d04be6252b-metrics-tls\") pod \"dns-default-cbr79\" (UID: \"3e262594-dced-40d6-b678-a0d04be6252b\") " pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:57.711344 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.711259    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 10:22:57.721522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.721499    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 10:22:57.731938 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.731912    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/cc37245a-69d3-4f71-9011-40a62c5457a2-kube-api-access-bst47\") pod \"network-check-target-m2fg8\" (UID: \"cc37245a-69d3-4f71-9011-40a62c5457a2\") " pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:57.809125 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.809084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:57.809323 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.809167    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:57.811627 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.811568    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 10:22:57.811755 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.811711    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 10:22:57.822189 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.822162    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d-original-pull-secret\") pod \"global-pull-secret-syncer-hpfqd\" (UID: \"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d\") " pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:57.822189 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.822179    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ec7fd0-b053-4c24-b252-a32a59409d59-metrics-certs\") pod \"network-metrics-daemon-w7hwz\" (UID: \"21ec7fd0-b053-4c24-b252-a32a59409d59\") " pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:57.938024 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.937993    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xrhk7\""
Sep 06 10:22:57.945477 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.945447    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:22:57.945617 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.945447    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xmc8p\""
Sep 06 10:22:57.949626 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.949597    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hpfqd"
Sep 06 10:22:57.953331 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.953313    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w7hwz"
Sep 06 10:22:57.975968 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.975894    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mhwb9\""
Sep 06 10:22:57.984059 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:57.984031    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cbr79"
Sep 06 10:22:58.875931 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:58.875857    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cbr79"]
Sep 06 10:22:58.878769 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:22:58.878731    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e262594_dced_40d6_b678_a0d04be6252b.slice/crio-60ff37fc99b2955b89d4ae7c566b55f39470be3e1e948827dc2e7547acb2e3a2 WatchSource:0}: Error finding container 60ff37fc99b2955b89d4ae7c566b55f39470be3e1e948827dc2e7547acb2e3a2: Status 404 returned error can't find the container with id 60ff37fc99b2955b89d4ae7c566b55f39470be3e1e948827dc2e7547acb2e3a2
Sep 06 10:22:58.881163 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:58.881083    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2fg8"]
Sep 06 10:22:58.885004 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:22:58.884981    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc37245a_69d3_4f71_9011_40a62c5457a2.slice/crio-5fcba7b11a726ca67e90df541b04525d9026bd493ff256beba5d25b89436ff27 WatchSource:0}: Error finding container 5fcba7b11a726ca67e90df541b04525d9026bd493ff256beba5d25b89436ff27: Status 404 returned error can't find the container with id 5fcba7b11a726ca67e90df541b04525d9026bd493ff256beba5d25b89436ff27
Sep 06 10:22:58.901824 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:58.901793    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w7hwz"]
Sep 06 10:22:58.904885 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:58.904861    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hpfqd"]
Sep 06 10:22:58.905696 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:22:58.905653    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ec7fd0_b053_4c24_b252_a32a59409d59.slice/crio-3d19cee80cae1d94eabe31e4350f6888c6bcfeb0b805e3666c75c05b60094150 WatchSource:0}: Error finding container 3d19cee80cae1d94eabe31e4350f6888c6bcfeb0b805e3666c75c05b60094150: Status 404 returned error can't find the container with id 3d19cee80cae1d94eabe31e4350f6888c6bcfeb0b805e3666c75c05b60094150
Sep 06 10:22:58.907519 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:22:58.907493    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a17bb93_98a4_4d0b_8f03_2c86b1f09b0d.slice/crio-c8224b24d140ec68f8aafcfc893739e38ccc69a57a93d36f258dbe33076836b9 WatchSource:0}: Error finding container c8224b24d140ec68f8aafcfc893739e38ccc69a57a93d36f258dbe33076836b9: Status 404 returned error can't find the container with id c8224b24d140ec68f8aafcfc893739e38ccc69a57a93d36f258dbe33076836b9
Sep 06 10:22:59.326841 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.326755    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w7hwz" event={"ID":"21ec7fd0-b053-4c24-b252-a32a59409d59","Type":"ContainerStarted","Data":"3d19cee80cae1d94eabe31e4350f6888c6bcfeb0b805e3666c75c05b60094150"}
Sep 06 10:22:59.328977 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.328945    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2fg8" event={"ID":"cc37245a-69d3-4f71-9011-40a62c5457a2","Type":"ContainerStarted","Data":"5fcba7b11a726ca67e90df541b04525d9026bd493ff256beba5d25b89436ff27"}
Sep 06 10:22:59.332999 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.332359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4" event={"ID":"57a1b94c-1988-43d2-a4c8-447ed0464d16","Type":"ContainerStarted","Data":"16698ab813d2d12c55ecae35d1d99b36e413f4dd1e2f24709f86add151fe97c5"}
Sep 06 10:22:59.336974 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.336904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cbr79" event={"ID":"3e262594-dced-40d6-b678-a0d04be6252b","Type":"ContainerStarted","Data":"60ff37fc99b2955b89d4ae7c566b55f39470be3e1e948827dc2e7547acb2e3a2"}
Sep 06 10:22:59.338667 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.338640    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hpfqd" event={"ID":"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d","Type":"ContainerStarted","Data":"c8224b24d140ec68f8aafcfc893739e38ccc69a57a93d36f258dbe33076836b9"}
Sep 06 10:22:59.350457 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.350052    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf6944c96-wtgr4" podStartSLOduration=1.965540507 podStartE2EDuration="4.350033216s" totalImagesPullingTime="2.384492709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:59.348112925 +0000 UTC m=+67.831613055" watchObservedRunningTime="2026-09-06 10:22:59.350033216 +0000 UTC m=+67.833533326"
Sep 06 10:22:59.861393 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.861153    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"]
Sep 06 10:22:59.868817 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.868770    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"
Sep 06 10:22:59.871839 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.871815    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 06 10:22:59.872047 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.872026    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6tqwk\""
Sep 06 10:22:59.880513 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.880491    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"]
Sep 06 10:22:59.926496 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.926383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ed020c3-6474-434e-b11d-c0ecb7f68e83-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-bjmmw\" (UID: \"0ed020c3-6474-434e-b11d-c0ecb7f68e83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"
Sep 06 10:22:59.961973 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.961930    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-h8bhl"]
Sep 06 10:22:59.966747 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.966720    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-h8bhl"
Sep 06 10:22:59.970617 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.970127    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-g86b2\""
Sep 06 10:22:59.970617 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.970266    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 06 10:22:59.971308 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.971111    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 06 10:22:59.980205 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:22:59.977349    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-h8bhl"]
Sep 06 10:23:00.026838 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.026807    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwlpj\" (UniqueName: \"kubernetes.io/projected/2549c291-d66f-4887-93ce-cc0ad7d761f7-kube-api-access-bwlpj\") pod \"downloads-85d98c998f-h8bhl\" (UID: \"2549c291-d66f-4887-93ce-cc0ad7d761f7\") " pod="openshift-console/downloads-85d98c998f-h8bhl"
Sep 06 10:23:00.027023 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.026862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ed020c3-6474-434e-b11d-c0ecb7f68e83-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-bjmmw\" (UID: \"0ed020c3-6474-434e-b11d-c0ecb7f68e83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"
Sep 06 10:23:00.027023 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:23:00.026997    2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 06 10:23:00.027463 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:23:00.027084    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ed020c3-6474-434e-b11d-c0ecb7f68e83-tls-certificates podName:0ed020c3-6474-434e-b11d-c0ecb7f68e83 nodeName:}" failed. No retries permitted until 2026-09-06 10:23:00.527061598 +0000 UTC m=+69.010561706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/0ed020c3-6474-434e-b11d-c0ecb7f68e83-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-bjmmw" (UID: "0ed020c3-6474-434e-b11d-c0ecb7f68e83") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 06 10:23:00.127613 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.127575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwlpj\" (UniqueName: \"kubernetes.io/projected/2549c291-d66f-4887-93ce-cc0ad7d761f7-kube-api-access-bwlpj\") pod \"downloads-85d98c998f-h8bhl\" (UID: \"2549c291-d66f-4887-93ce-cc0ad7d761f7\") " pod="openshift-console/downloads-85d98c998f-h8bhl"
Sep 06 10:23:00.140191 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.140083    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwlpj\" (UniqueName: \"kubernetes.io/projected/2549c291-d66f-4887-93ce-cc0ad7d761f7-kube-api-access-bwlpj\") pod \"downloads-85d98c998f-h8bhl\" (UID: \"2549c291-d66f-4887-93ce-cc0ad7d761f7\") " pod="openshift-console/downloads-85d98c998f-h8bhl"
Sep 06 10:23:00.280573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.280332    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-h8bhl"
Sep 06 10:23:00.351171 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.350711    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w7hwz" event={"ID":"21ec7fd0-b053-4c24-b252-a32a59409d59","Type":"ContainerStarted","Data":"d5547f7575b3967c22961664c450aa0e2b6e0e2199b79db0045057d00d3283bf"}
Sep 06 10:23:00.351171 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.350768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w7hwz" event={"ID":"21ec7fd0-b053-4c24-b252-a32a59409d59","Type":"ContainerStarted","Data":"aa33dff5fd0d8caafb5046bf8e6c249dd9b61aa7bb00160414708bf9026b036b"}
Sep 06 10:23:00.531929 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.531351    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ed020c3-6474-434e-b11d-c0ecb7f68e83-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-bjmmw\" (UID: \"0ed020c3-6474-434e-b11d-c0ecb7f68e83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"
Sep 06 10:23:00.535170 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.535115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ed020c3-6474-434e-b11d-c0ecb7f68e83-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-bjmmw\" (UID: \"0ed020c3-6474-434e-b11d-c0ecb7f68e83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"
Sep 06 10:23:00.582933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.582068    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w7hwz" podStartSLOduration=67.459976216 podStartE2EDuration="1m8.582041812s" totalImagesPullingTime="1.122065596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:00.374639655 +0000 UTC m=+68.858139790" watchObservedRunningTime="2026-09-06 10:23:00.582041812 +0000 UTC m=+69.065541922"
Sep 06 10:23:00.582933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.582926    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-h8bhl"]
Sep 06 10:23:00.782472 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:00.782260    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"
Sep 06 10:23:01.076713 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:23:01.076664    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2549c291_d66f_4887_93ce_cc0ad7d761f7.slice/crio-361734629fac6422660a9bc84173a8a84d6c247f38af07bce335eeaf580e91f5 WatchSource:0}: Error finding container 361734629fac6422660a9bc84173a8a84d6c247f38af07bce335eeaf580e91f5: Status 404 returned error can't find the container with id 361734629fac6422660a9bc84173a8a84d6c247f38af07bce335eeaf580e91f5
Sep 06 10:23:01.357229 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:01.357189    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"]
Sep 06 10:23:01.361736 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:01.361673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cbr79" event={"ID":"3e262594-dced-40d6-b678-a0d04be6252b","Type":"ContainerStarted","Data":"24a68467bc26608c8c11a17a471bf90157fc28d851e4d62012fc334a6cb982fc"}
Sep 06 10:23:01.363524 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:01.363458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-h8bhl" event={"ID":"2549c291-d66f-4887-93ce-cc0ad7d761f7","Type":"ContainerStarted","Data":"361734629fac6422660a9bc84173a8a84d6c247f38af07bce335eeaf580e91f5"}
Sep 06 10:23:02.281202 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:23:02.281123    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed020c3_6474_434e_b11d_c0ecb7f68e83.slice/crio-74f9d2da77567dbba36ad0ca4bddbdb09220dffe657467c48700c25647e2ecd5 WatchSource:0}: Error finding container 74f9d2da77567dbba36ad0ca4bddbdb09220dffe657467c48700c25647e2ecd5: Status 404 returned error can't find the container with id 74f9d2da77567dbba36ad0ca4bddbdb09220dffe657467c48700c25647e2ecd5
Sep 06 10:23:02.370973 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:02.370933    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw" event={"ID":"0ed020c3-6474-434e-b11d-c0ecb7f68e83","Type":"ContainerStarted","Data":"74f9d2da77567dbba36ad0ca4bddbdb09220dffe657467c48700c25647e2ecd5"}
Sep 06 10:23:04.381177 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:04.381132    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hpfqd" event={"ID":"6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d","Type":"ContainerStarted","Data":"fb8e72345d3b323abf56c87cb6d019b675ed4b9152b51bfe5654385ebfd4ab88"}
Sep 06 10:23:04.383783 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:04.383704    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2fg8" event={"ID":"cc37245a-69d3-4f71-9011-40a62c5457a2","Type":"ContainerStarted","Data":"cd4a3103728d1d54c4dcff540b902a5be3cf8146c04571775213114054fee37d"}
Sep 06 10:23:04.383921 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:04.383802    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:23:04.385998 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:04.385972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cbr79" event={"ID":"3e262594-dced-40d6-b678-a0d04be6252b","Type":"ContainerStarted","Data":"3d157c203de97e4e4484762c0f93b62716b4e9c85ecf10bafeb74dc8cd053b6b"}
Sep 06 10:23:04.386126 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:04.386110    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cbr79"
Sep 06 10:23:04.397187 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:04.397127    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hpfqd" podStartSLOduration=67.385834079 podStartE2EDuration="1m12.397107197s" totalImagesPullingTime="5.011273118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:04.395499278 +0000 UTC m=+72.878999401" watchObservedRunningTime="2026-09-06 10:23:04.397107197 +0000 UTC m=+72.880607307"
Sep 06 10:23:04.411751 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:04.411685    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cbr79" podStartSLOduration=37.151016412 podStartE2EDuration="39.411663098s" totalImagesPullingTime="2.260646686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:04.410944054 +0000 UTC m=+72.894444192" watchObservedRunningTime="2026-09-06 10:23:04.411663098 +0000 UTC m=+72.895163208"
Sep 06 10:23:04.425631 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:04.425566    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m2fg8" podStartSLOduration=67.400324748 podStartE2EDuration="1m12.425540901s" totalImagesPullingTime="5.025216153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:04.425282042 +0000 UTC m=+72.908782154" watchObservedRunningTime="2026-09-06 10:23:04.425540901 +0000 UTC m=+72.909041011"
Sep 06 10:23:05.392139 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:05.392060    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw" event={"ID":"0ed020c3-6474-434e-b11d-c0ecb7f68e83","Type":"ContainerStarted","Data":"bed655d8e683e5afda75f1e45fb653376d2649d37e6ec053ec56de556f809116"}
Sep 06 10:23:05.392618 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:05.392265    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"
Sep 06 10:23:05.397744 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:05.397718    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw"
Sep 06 10:23:05.407349 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:05.407189    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bjmmw" podStartSLOduration=4.036424105 podStartE2EDuration="6.407172086s" totalImagesPullingTime="2.370747981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:05.407101611 +0000 UTC m=+73.890601719" watchObservedRunningTime="2026-09-06 10:23:05.407172086 +0000 UTC m=+73.890672217"
Sep 06 10:23:10.280977 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.280940    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"]
Sep 06 10:23:10.292121 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.291636    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.295639 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.295569    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"]
Sep 06 10:23:10.296092 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.296071    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xwtvp\""
Sep 06 10:23:10.296324 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.296309    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 06 10:23:10.297058 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.296507    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 06 10:23:10.297058 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.296711    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 06 10:23:10.297058 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.296905    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 06 10:23:10.297291 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.297069    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 06 10:23:10.305461 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.300014    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-brshs"]
Sep 06 10:23:10.305461 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.304244    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.309512 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.307343    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nl6j2\""
Sep 06 10:23:10.309512 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.307658    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 06 10:23:10.309512 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.307827    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 06 10:23:10.309512 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.307990    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 06 10:23:10.309512 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.308147    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n72xw"]
Sep 06 10:23:10.313178 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.312639    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.315894 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.315497    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 06 10:23:10.315894 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.315744    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fsv89\""
Sep 06 10:23:10.315894 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.315774    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 06 10:23:10.315894 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.315842    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 06 10:23:10.337569 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.337434    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-brshs"]
Sep 06 10:23:10.415127 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.414932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69p76\" (UniqueName: \"kubernetes.io/projected/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-api-access-69p76\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.415127 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.414989    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.415127 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415023    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-tls\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.415127 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415053    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48d31127-0fb1-4758-91b6-26f51b39b5c8-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415132    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/48d31127-0fb1-4758-91b6-26f51b39b5c8-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415176    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-textfile\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415246    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-wtmp\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415272    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vb8x\" (UniqueName: \"kubernetes.io/projected/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-kube-api-access-6vb8x\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415298    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc98d\" (UniqueName: \"kubernetes.io/projected/a91dff9a-5731-4beb-93a8-a0f6efb80dad-kube-api-access-gc98d\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415353    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.415467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415452    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a91dff9a-5731-4beb-93a8-a0f6efb80dad-root\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.415984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415478    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a91dff9a-5731-4beb-93a8-a0f6efb80dad-sys\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.415984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a91dff9a-5731-4beb-93a8-a0f6efb80dad-metrics-client-ca\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.415984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.415984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.415555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.516135 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516094    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.516340 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516152    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69p76\" (UniqueName: \"kubernetes.io/projected/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-api-access-69p76\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.516340 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.516340 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-tls\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.516340 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48d31127-0fb1-4758-91b6-26f51b39b5c8-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.516340 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/48d31127-0fb1-4758-91b6-26f51b39b5c8-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.516340 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-textfile\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516348    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-wtmp\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516403    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vb8x\" (UniqueName: \"kubernetes.io/projected/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-kube-api-access-6vb8x\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516427    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc98d\" (UniqueName: \"kubernetes.io/projected/a91dff9a-5731-4beb-93a8-a0f6efb80dad-kube-api-access-gc98d\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a91dff9a-5731-4beb-93a8-a0f6efb80dad-root\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a91dff9a-5731-4beb-93a8-a0f6efb80dad-sys\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.516642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516636    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a91dff9a-5731-4beb-93a8-a0f6efb80dad-metrics-client-ca\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.517152 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.517152 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516760    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/48d31127-0fb1-4758-91b6-26f51b39b5c8-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.517152 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:23:10.516790    2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 06 10:23:10.517152 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:23:10.516861    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-openshift-state-metrics-tls podName:250c73c6-2b37-4b8c-90a7-5f8ae4c23c93 nodeName:}" failed. No retries permitted until 2026-09-06 10:23:11.016840789 +0000 UTC m=+79.500340879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-pwwkl" (UID: "250c73c6-2b37-4b8c-90a7-5f8ae4c23c93") : secret "openshift-state-metrics-tls" not found
Sep 06 10:23:10.517152 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.516965    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.517437 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.517269    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-textfile\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.517493 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:23:10.517467    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 06 10:23:10.517551 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:23:10.517537    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-tls podName:a91dff9a-5731-4beb-93a8-a0f6efb80dad nodeName:}" failed. No retries permitted until 2026-09-06 10:23:11.017518118 +0000 UTC m=+79.501018221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-tls") pod "node-exporter-n72xw" (UID: "a91dff9a-5731-4beb-93a8-a0f6efb80dad") : secret "node-exporter-tls" not found
Sep 06 10:23:10.519047 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.517760    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-wtmp\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.519047 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.518451    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48d31127-0fb1-4758-91b6-26f51b39b5c8-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.519047 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.518512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a91dff9a-5731-4beb-93a8-a0f6efb80dad-sys\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.519047 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.518523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.519047 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.518783    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a91dff9a-5731-4beb-93a8-a0f6efb80dad-root\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.519047 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.519011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.519453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.519286    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a91dff9a-5731-4beb-93a8-a0f6efb80dad-metrics-client-ca\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.520928 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.520902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.521709 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.521679    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.522153 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.522126    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.523605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.523559    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.530151 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.530121    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69p76\" (UniqueName: \"kubernetes.io/projected/48d31127-0fb1-4758-91b6-26f51b39b5c8-kube-api-access-69p76\") pod \"kube-state-metrics-74cb57df7-brshs\" (UID: \"48d31127-0fb1-4758-91b6-26f51b39b5c8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.532173 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.532113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vb8x\" (UniqueName: \"kubernetes.io/projected/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-kube-api-access-6vb8x\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:10.532789 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.532747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc98d\" (UniqueName: \"kubernetes.io/projected/a91dff9a-5731-4beb-93a8-a0f6efb80dad-kube-api-access-gc98d\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:10.625386 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.625348    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs"
Sep 06 10:23:10.915558 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:10.915528    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-brshs"]
Sep 06 10:23:10.918110 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:23:10.918074    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d31127_0fb1_4758_91b6_26f51b39b5c8.slice/crio-7fc80d5230cf4a3cebf8eb157fca263cb8c0de3a6d0b58e3e5ff272375f3d80a WatchSource:0}: Error finding container 7fc80d5230cf4a3cebf8eb157fca263cb8c0de3a6d0b58e3e5ff272375f3d80a: Status 404 returned error can't find the container with id 7fc80d5230cf4a3cebf8eb157fca263cb8c0de3a6d0b58e3e5ff272375f3d80a
Sep 06 10:23:11.021778 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:11.021738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:11.021953 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:11.021809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-tls\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:11.021953 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:23:11.021929    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 06 10:23:11.022072 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:23:11.022000    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-tls podName:a91dff9a-5731-4beb-93a8-a0f6efb80dad nodeName:}" failed. No retries permitted until 2026-09-06 10:23:12.021980073 +0000 UTC m=+80.505480178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-tls") pod "node-exporter-n72xw" (UID: "a91dff9a-5731-4beb-93a8-a0f6efb80dad") : secret "node-exporter-tls" not found
Sep 06 10:23:11.024484 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:11.024460    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/250c73c6-2b37-4b8c-90a7-5f8ae4c23c93-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pwwkl\" (UID: \"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:11.210311 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:11.210206    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"
Sep 06 10:23:11.395320 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:11.395292    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cbr79"
Sep 06 10:23:11.420261 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:11.420224    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs" event={"ID":"48d31127-0fb1-4758-91b6-26f51b39b5c8","Type":"ContainerStarted","Data":"7fc80d5230cf4a3cebf8eb157fca263cb8c0de3a6d0b58e3e5ff272375f3d80a"}
Sep 06 10:23:11.495739 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:11.495601    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl"]
Sep 06 10:23:11.499631 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:23:11.499600    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250c73c6_2b37_4b8c_90a7_5f8ae4c23c93.slice/crio-0e54bb39571b6f36833dfc4737643a74b6d1fdc1d3d75f0b5c88a3fa597ff893 WatchSource:0}: Error finding container 0e54bb39571b6f36833dfc4737643a74b6d1fdc1d3d75f0b5c88a3fa597ff893: Status 404 returned error can't find the container with id 0e54bb39571b6f36833dfc4737643a74b6d1fdc1d3d75f0b5c88a3fa597ff893
Sep 06 10:23:12.029038 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.029007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-tls\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:12.031645 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.031617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a91dff9a-5731-4beb-93a8-a0f6efb80dad-node-exporter-tls\") pod \"node-exporter-n72xw\" (UID: \"a91dff9a-5731-4beb-93a8-a0f6efb80dad\") " pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:12.138515 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.138480    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n72xw"
Sep 06 10:23:12.268524 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.268358    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-56d9844b69-9w62x"]
Sep 06 10:23:12.273721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.273695    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.276275 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.276236    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 06 10:23:12.276518 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.276494    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 06 10:23:12.276607 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.276567    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l8r7f\""
Sep 06 10:23:12.276607 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.276574    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 06 10:23:12.276748 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.276720    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 06 10:23:12.276846 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.276772    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3q2g1qr8am0gj\""
Sep 06 10:23:12.276846 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.276772    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 06 10:23:12.281544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.281474    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56d9844b69-9w62x"]
Sep 06 10:23:12.426292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.426248    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl" event={"ID":"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93","Type":"ContainerStarted","Data":"00b5e69e74f85083cf5f71addd8166211760aa272080684d7a9cdd6914002b59"}
Sep 06 10:23:12.426701 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.426329    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl" event={"ID":"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93","Type":"ContainerStarted","Data":"bcbd277040a8e54c91883cbd96951764b9ad8cfa2d30e1cfdc8f83a7bc90a152"}
Sep 06 10:23:12.426701 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.426362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl" event={"ID":"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93","Type":"ContainerStarted","Data":"0e54bb39571b6f36833dfc4737643a74b6d1fdc1d3d75f0b5c88a3fa597ff893"}
Sep 06 10:23:12.431564 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.431531    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.431564 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.431568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-metrics-client-ca\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.431752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.431586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-grpc-tls\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.431752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.431608    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.431752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.431679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.431752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.431747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.431919 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.431782    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9xr2\" (UniqueName: \"kubernetes.io/projected/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-kube-api-access-q9xr2\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.431919 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.431839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-tls\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.533055 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.532962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.533055 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.533005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-metrics-client-ca\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.533055 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.533023    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-grpc-tls\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.533307 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.533255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.533357 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.533316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.533357 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.533350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.533470 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.533379    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9xr2\" (UniqueName: \"kubernetes.io/projected/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-kube-api-access-q9xr2\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.533470 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.533407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-tls\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.533839 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.533813    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-metrics-client-ca\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.536034 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.535984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.536034 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.536009    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.536394 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.536369    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-grpc-tls\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.536494 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.536369    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.536722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.536701    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-tls\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.536825 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.536807    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.542538 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.542520    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9xr2\" (UniqueName: \"kubernetes.io/projected/d6c3cfad-e5b4-4380-9c1e-286cc8dd1373-kube-api-access-q9xr2\") pod \"thanos-querier-56d9844b69-9w62x\" (UID: \"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373\") " pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:12.585588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:12.585544    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:14.744995 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.744859    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6c5874989c-ddjz6"]
Sep 06 10:23:14.750973 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.750951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.753717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.753692    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 06 10:23:14.754850 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.754814    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 06 10:23:14.754987 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.754818    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 06 10:23:14.754987 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.754954    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2a9tce2nn5k0j\""
Sep 06 10:23:14.754987 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.754960    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 06 10:23:14.755162 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.755091    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-774p5\""
Sep 06 10:23:14.760469 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.760447    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c5874989c-ddjz6"]
Sep 06 10:23:14.855800 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.855765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkn47\" (UniqueName: \"kubernetes.io/projected/09208b3f-f2a1-450c-a747-5316c8296be3-kube-api-access-jkn47\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.856000 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.855824    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09208b3f-f2a1-450c-a747-5316c8296be3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.856000 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.855862    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/09208b3f-f2a1-450c-a747-5316c8296be3-metrics-server-audit-profiles\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.856000 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.855896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/09208b3f-f2a1-450c-a747-5316c8296be3-client-ca-bundle\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.856000 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.855932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/09208b3f-f2a1-450c-a747-5316c8296be3-secret-metrics-server-tls\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.856000 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.855959    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/09208b3f-f2a1-450c-a747-5316c8296be3-secret-metrics-server-client-certs\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.856000 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.855993    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/09208b3f-f2a1-450c-a747-5316c8296be3-audit-log\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.956766 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.956726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09208b3f-f2a1-450c-a747-5316c8296be3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.956961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.956783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/09208b3f-f2a1-450c-a747-5316c8296be3-metrics-server-audit-profiles\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.956961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.956816    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/09208b3f-f2a1-450c-a747-5316c8296be3-client-ca-bundle\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.956961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.956848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/09208b3f-f2a1-450c-a747-5316c8296be3-secret-metrics-server-tls\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.956961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.956879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/09208b3f-f2a1-450c-a747-5316c8296be3-secret-metrics-server-client-certs\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.956961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.956916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/09208b3f-f2a1-450c-a747-5316c8296be3-audit-log\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.956961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.956952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkn47\" (UniqueName: \"kubernetes.io/projected/09208b3f-f2a1-450c-a747-5316c8296be3-kube-api-access-jkn47\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.957671 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.957594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09208b3f-f2a1-450c-a747-5316c8296be3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.957671 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.957624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/09208b3f-f2a1-450c-a747-5316c8296be3-audit-log\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.957943 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.957918    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/09208b3f-f2a1-450c-a747-5316c8296be3-metrics-server-audit-profiles\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.959907 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.959881    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/09208b3f-f2a1-450c-a747-5316c8296be3-client-ca-bundle\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.960035 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.959919    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/09208b3f-f2a1-450c-a747-5316c8296be3-secret-metrics-server-client-certs\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.960232 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.960195    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/09208b3f-f2a1-450c-a747-5316c8296be3-secret-metrics-server-tls\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:14.964887 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:14.964867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkn47\" (UniqueName: \"kubernetes.io/projected/09208b3f-f2a1-450c-a747-5316c8296be3-kube-api-access-jkn47\") pod \"metrics-server-6c5874989c-ddjz6\" (UID: \"09208b3f-f2a1-450c-a747-5316c8296be3\") " pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:15.061235 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.061181    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"]
Sep 06 10:23:15.063002 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.062974    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:15.066899 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.066876    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:15.069637 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.069499    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 06 10:23:15.069637 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.069613    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-44sf6\""
Sep 06 10:23:15.083541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.083512    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"]
Sep 06 10:23:15.158717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.158679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h76j2\" (UniqueName: \"kubernetes.io/projected/9a981f69-c807-4a4e-85a6-0f8deb6f5e31-kube-api-access-h76j2\") pod \"monitoring-plugin-85bbbbbf4-rcdvj\" (UID: \"9a981f69-c807-4a4e-85a6-0f8deb6f5e31\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:15.158916 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.158746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a981f69-c807-4a4e-85a6-0f8deb6f5e31-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rcdvj\" (UID: \"9a981f69-c807-4a4e-85a6-0f8deb6f5e31\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:15.259860 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.259825    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76j2\" (UniqueName: \"kubernetes.io/projected/9a981f69-c807-4a4e-85a6-0f8deb6f5e31-kube-api-access-h76j2\") pod \"monitoring-plugin-85bbbbbf4-rcdvj\" (UID: \"9a981f69-c807-4a4e-85a6-0f8deb6f5e31\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:15.260035 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.259878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a981f69-c807-4a4e-85a6-0f8deb6f5e31-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rcdvj\" (UID: \"9a981f69-c807-4a4e-85a6-0f8deb6f5e31\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:15.260035 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:23:15.259994    2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 06 10:23:15.260117 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:23:15.260085    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a981f69-c807-4a4e-85a6-0f8deb6f5e31-monitoring-plugin-cert podName:9a981f69-c807-4a4e-85a6-0f8deb6f5e31 nodeName:}" failed. No retries permitted until 2026-09-06 10:23:15.760064211 +0000 UTC m=+84.243564306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9a981f69-c807-4a4e-85a6-0f8deb6f5e31-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-rcdvj" (UID: "9a981f69-c807-4a4e-85a6-0f8deb6f5e31") : secret "monitoring-plugin-cert" not found
Sep 06 10:23:15.271135 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.271107    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h76j2\" (UniqueName: \"kubernetes.io/projected/9a981f69-c807-4a4e-85a6-0f8deb6f5e31-kube-api-access-h76j2\") pod \"monitoring-plugin-85bbbbbf4-rcdvj\" (UID: \"9a981f69-c807-4a4e-85a6-0f8deb6f5e31\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:15.765041 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.764994    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a981f69-c807-4a4e-85a6-0f8deb6f5e31-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rcdvj\" (UID: \"9a981f69-c807-4a4e-85a6-0f8deb6f5e31\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:15.767820 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.767792    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a981f69-c807-4a4e-85a6-0f8deb6f5e31-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rcdvj\" (UID: \"9a981f69-c807-4a4e-85a6-0f8deb6f5e31\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:15.979230 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:15.979179    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:16.083139 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.083079    2564 patch_prober.go:28] interesting pod/image-registry-6f64dddb67-t4fz6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 06 10:23:16.083338 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.083169    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6" podUID="71bd4b2d-5317-4a86-9bbf-9995f3b667f2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 10:23:16.465547 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.465464    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:23:16.469838 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.469811    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.472637 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.472610    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 06 10:23:16.472779 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.472610    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 06 10:23:16.472884 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.472862    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 06 10:23:16.472948 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.472907    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 06 10:23:16.472999 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.472975    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 06 10:23:16.473051 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.472994    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 06 10:23:16.473135 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.473116    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 06 10:23:16.473199 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.473149    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-71r72ghhmhfc4\""
Sep 06 10:23:16.473650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.473622    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 06 10:23:16.473650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.473638    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 06 10:23:16.473814 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.473708    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 06 10:23:16.473814 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.473728    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 06 10:23:16.473814 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.473734    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 06 10:23:16.473978 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.473957    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 06 10:23:16.473978 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.473969    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4frx7\""
Sep 06 10:23:16.476764 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.476746    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 06 10:23:16.488854 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.488831    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:23:16.573138 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573101    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573138 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573139    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573399 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573399 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573399 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573284    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573399 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573399 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573324    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-web-config\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573471    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573501    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573535    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573589    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573613    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-config\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.573660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573652    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.574097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmvv\" (UniqueName: \"kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-kube-api-access-dgmvv\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.574097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573731    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.574097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-config-out\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.574097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.573770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675152 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675152 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-web-config\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675467    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675489    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-config\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmvv\" (UniqueName: \"kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-kube-api-access-dgmvv\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675642    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-config-out\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.675756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.675659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.676721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.676412    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.676721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.676456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.676721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.676633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.676975 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.676929    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.678158 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.677809    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.679781 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.678937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.679781 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.679190    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.679781 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.679462    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.679996 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.679796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.679996 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.679913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-web-config\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.680097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.680035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.681535 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.681508    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.681763 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.681733    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.682064 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.682039    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.682453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.682425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.682571 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.682550    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-config\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.682665 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.682645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-config-out\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.683542 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.683518    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.684439 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.684417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.696409 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.696389    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmvv\" (UniqueName: \"kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-kube-api-access-dgmvv\") pod \"prometheus-k8s-0\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:16.782244 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:16.782141    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:18.323460 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:18.323430    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f64dddb67-t4fz6"
Sep 06 10:23:19.540674 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:23:19.540610    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91dff9a_5731_4beb_93a8_a0f6efb80dad.slice/crio-081624c94a389863c590b6d7255e534b871a991c9f770664e87a909d6cf8d5e5 WatchSource:0}: Error finding container 081624c94a389863c590b6d7255e534b871a991c9f770664e87a909d6cf8d5e5: Status 404 returned error can't find the container with id 081624c94a389863c590b6d7255e534b871a991c9f770664e87a909d6cf8d5e5
Sep 06 10:23:19.840239 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:19.838880    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"]
Sep 06 10:23:19.843248 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:19.843132    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c5874989c-ddjz6"]
Sep 06 10:23:19.847129 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:19.847093    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56d9844b69-9w62x"]
Sep 06 10:23:19.866931 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:19.866900    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:23:19.873562 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:23:19.873506    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44e8a25_cbbf_4c45_b727_30bac91f7103.slice/crio-1112d78cbda3cd7c77f826ce7fa20fffdb6cce27a5ba4964639fcc1dd0f037d7 WatchSource:0}: Error finding container 1112d78cbda3cd7c77f826ce7fa20fffdb6cce27a5ba4964639fcc1dd0f037d7: Status 404 returned error can't find the container with id 1112d78cbda3cd7c77f826ce7fa20fffdb6cce27a5ba4964639fcc1dd0f037d7
Sep 06 10:23:20.469554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.469470    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs" event={"ID":"48d31127-0fb1-4758-91b6-26f51b39b5c8","Type":"ContainerStarted","Data":"8d03254cf5e42126603062acd7fd40ad536fe10db09df3e5be9e88be1f475afb"}
Sep 06 10:23:20.470277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.469755    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs" event={"ID":"48d31127-0fb1-4758-91b6-26f51b39b5c8","Type":"ContainerStarted","Data":"f6a82dde4930b6853251855e43118698882c98375c706377e19c712fb07e13ce"}
Sep 06 10:23:20.470277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.469816    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs" event={"ID":"48d31127-0fb1-4758-91b6-26f51b39b5c8","Type":"ContainerStarted","Data":"82dff46835146a50531c835dc9d544b42893484d48215d7116ce8a3c710c6f26"}
Sep 06 10:23:20.473910 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.473825    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n72xw" event={"ID":"a91dff9a-5731-4beb-93a8-a0f6efb80dad","Type":"ContainerStarted","Data":"081624c94a389863c590b6d7255e534b871a991c9f770664e87a909d6cf8d5e5"}
Sep 06 10:23:20.475696 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.475605    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x" event={"ID":"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373","Type":"ContainerStarted","Data":"e1f4d8dbd118d8ca932e0b650a10c7899d370cf596c89d3d5a9553cbb0c605a3"}
Sep 06 10:23:20.478650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.478625    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerStarted","Data":"1112d78cbda3cd7c77f826ce7fa20fffdb6cce27a5ba4964639fcc1dd0f037d7"}
Sep 06 10:23:20.480895 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.480835    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6" event={"ID":"09208b3f-f2a1-450c-a747-5316c8296be3","Type":"ContainerStarted","Data":"2a1ce0bdf5106806f75b829520ab40811932fd6874b15df43d303c144094daa9"}
Sep 06 10:23:20.484336 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.484172    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-h8bhl" event={"ID":"2549c291-d66f-4887-93ce-cc0ad7d761f7","Type":"ContainerStarted","Data":"57bc760117e713242b55d72000d0d52734b45353f9eace8bbe51c0a57d2b995b"}
Sep 06 10:23:20.485233 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.485140    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-h8bhl"
Sep 06 10:23:20.486635 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.486570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj" event={"ID":"9a981f69-c807-4a4e-85a6-0f8deb6f5e31","Type":"ContainerStarted","Data":"c958b4dfa82db36685fcb58321ef773c5be59697c5c674a95352cee52fbfb821"}
Sep 06 10:23:20.499745 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.499692    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-h8bhl"
Sep 06 10:23:20.521446 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.521233    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-h8bhl" podStartSLOduration=2.995319865 podStartE2EDuration="21.521197225s" totalImagesPullingTime="18.52587736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:20.521009868 +0000 UTC m=+89.004509978" watchObservedRunningTime="2026-09-06 10:23:20.521197225 +0000 UTC m=+89.004697334"
Sep 06 10:23:20.522995 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:20.522951    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-brshs" podStartSLOduration=1.912274381 podStartE2EDuration="10.522935467s" totalImagesPullingTime="8.610661086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:20.495868666 +0000 UTC m=+88.979368793" watchObservedRunningTime="2026-09-06 10:23:20.522935467 +0000 UTC m=+89.006435578"
Sep 06 10:23:21.497849 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:21.497816    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl" event={"ID":"250c73c6-2b37-4b8c-90a7-5f8ae4c23c93","Type":"ContainerStarted","Data":"3bbafd82972dbe68ceb641c788e370e14de26621b92bbc6f144b592258c3413f"}
Sep 06 10:23:25.520354 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.520320    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x" event={"ID":"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373","Type":"ContainerStarted","Data":"13c9fc51c909db42b060c9c9b3ace8ff8b38c0221a0f80558dbf43700d5623d0"}
Sep 06 10:23:25.520823 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.520406    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x" event={"ID":"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373","Type":"ContainerStarted","Data":"7bfc26dc851990a27c8b17d8b7b318c46632521edf9c605ebfd0369f6d94c0cf"}
Sep 06 10:23:25.520823 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.520452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x" event={"ID":"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373","Type":"ContainerStarted","Data":"7a5a09972bc760e723356df60411ef662fd2a0b0b9416abf593ad9b65743c33a"}
Sep 06 10:23:25.522293 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.522262    2564 generic.go:356] "Generic (PLEG): container finished" podID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerID="63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb" exitCode=0
Sep 06 10:23:25.522398 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.522350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerDied","Data":"63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb"}
Sep 06 10:23:25.524357 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.524326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6" event={"ID":"09208b3f-f2a1-450c-a747-5316c8296be3","Type":"ContainerStarted","Data":"7f2e7eecdbfe97d955af5375eae0e9a22565f100f3414d6bb11b61b68903f2a1"}
Sep 06 10:23:25.526480 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.526454    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj" event={"ID":"9a981f69-c807-4a4e-85a6-0f8deb6f5e31","Type":"ContainerStarted","Data":"c54be5c70309caaa4ee08194cc04a8c85e2a839effb0eb518b23b6a988c82ae4"}
Sep 06 10:23:25.526707 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.526687    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:25.528814 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.528788    2564 generic.go:356] "Generic (PLEG): container finished" podID="a91dff9a-5731-4beb-93a8-a0f6efb80dad" containerID="8173581d6cb32381969752022a4f52b6cf19fcf7cee6f32edf13df48610a6500" exitCode=0
Sep 06 10:23:25.528949 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.528825    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n72xw" event={"ID":"a91dff9a-5731-4beb-93a8-a0f6efb80dad","Type":"ContainerDied","Data":"8173581d6cb32381969752022a4f52b6cf19fcf7cee6f32edf13df48610a6500"}
Sep 06 10:23:25.532779 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.532757    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj"
Sep 06 10:23:25.550890 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.550836    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pwwkl" podStartSLOduration=6.872827791 podStartE2EDuration="15.550819335s" totalImagesPullingTime="8.677991544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:21.520901727 +0000 UTC m=+90.004401862" watchObservedRunningTime="2026-09-06 10:23:25.550819335 +0000 UTC m=+94.034319445"
Sep 06 10:23:25.567553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.567501    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rcdvj" podStartSLOduration=5.791900626 podStartE2EDuration="10.567482923s" totalImagesPullingTime="4.775582297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:25.566650955 +0000 UTC m=+94.050151067" watchObservedRunningTime="2026-09-06 10:23:25.567482923 +0000 UTC m=+94.050983032"
Sep 06 10:23:25.588252 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:25.588177    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6" podStartSLOduration=6.821402693 podStartE2EDuration="11.588154259s" totalImagesPullingTime="4.766751566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:25.586600889 +0000 UTC m=+94.070101032" watchObservedRunningTime="2026-09-06 10:23:25.588154259 +0000 UTC m=+94.071654370"
Sep 06 10:23:26.537379 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:26.536721    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n72xw" event={"ID":"a91dff9a-5731-4beb-93a8-a0f6efb80dad","Type":"ContainerStarted","Data":"62fd794ed98f0df7359cdc5c8fcb628c316295b9af9c026bfb8256601f3c8e00"}
Sep 06 10:23:26.537379 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:26.536797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n72xw" event={"ID":"a91dff9a-5731-4beb-93a8-a0f6efb80dad","Type":"ContainerStarted","Data":"44010b0b5b3b5fcadddf3dff2dfbffd6525a09b02d84781809213800900b34e2"}
Sep 06 10:23:26.565402 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:26.565335    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n72xw" podStartSLOduration=11.691629154 podStartE2EDuration="16.565314865s" totalImagesPullingTime="4.784186461s" totalInitContainerRuntime="89.49925ms" isStatefulPod=true podCreationTimestamp="2026-09-06 10:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:26.563287123 +0000 UTC m=+95.046787262" watchObservedRunningTime="2026-09-06 10:23:26.565314865 +0000 UTC m=+95.048814986"
Sep 06 10:23:27.547900 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:27.547771    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x" event={"ID":"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373","Type":"ContainerStarted","Data":"584d30611f86d3c235d9c0b2e9738acc1ff6aa775919daea7a82e684f0a26e00"}
Sep 06 10:23:27.547900 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:27.547853    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x" event={"ID":"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373","Type":"ContainerStarted","Data":"683a2ca7e1727c5a708e196e86e9268f0ace082b0c01cf36521d6dcb8b0892a5"}
Sep 06 10:23:27.547900 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:27.547900    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:27.548637 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:27.547954    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x" event={"ID":"d6c3cfad-e5b4-4380-9c1e-286cc8dd1373","Type":"ContainerStarted","Data":"29e69948af89df6727df324d185c13a4966598e0b126a9847c6485c4a4aca385"}
Sep 06 10:23:27.581875 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:27.581816    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x" podStartSLOduration=9.432577069 podStartE2EDuration="15.581791353s" totalImagesPullingTime="6.149214284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:23:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:27.579303178 +0000 UTC m=+96.062803327" watchObservedRunningTime="2026-09-06 10:23:27.581791353 +0000 UTC m=+96.065291463"
Sep 06 10:23:30.564199 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:30.564164    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerStarted","Data":"6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898"}
Sep 06 10:23:30.564708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:30.564310    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerStarted","Data":"17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2"}
Sep 06 10:23:30.564708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:30.564364    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerStarted","Data":"10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0"}
Sep 06 10:23:30.564708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:30.564409    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerStarted","Data":"c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a"}
Sep 06 10:23:30.564708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:30.564450    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerStarted","Data":"e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f"}
Sep 06 10:23:30.564708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:30.564489    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerStarted","Data":"79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0"}
Sep 06 10:23:30.594777 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:30.594722    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.692054026 podStartE2EDuration="14.594705655s" totalImagesPullingTime="8.879155362s" totalInitContainerRuntime="23.496267ms" isStatefulPod=false podCreationTimestamp="2026-09-06 10:23:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:30.593003963 +0000 UTC m=+99.076504091" watchObservedRunningTime="2026-09-06 10:23:30.594705655 +0000 UTC m=+99.078205764"
Sep 06 10:23:31.782486 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:31.782449    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:23:33.560076 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:33.560046    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-56d9844b69-9w62x"
Sep 06 10:23:35.063530 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:35.063493    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:35.063990 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:35.063553    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:35.394828 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:35.394744    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m2fg8"
Sep 06 10:23:55.069123 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:55.069094    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:23:55.072930 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:23:55.072910    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6c5874989c-ddjz6"
Sep 06 10:24:16.782426 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:16.782381    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:16.803664 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:16.803636    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:17.773648 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:17.773618    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:34.863551 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:34.863519    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:24:34.864044 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:34.863789    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="prometheus" containerID="cri-o://79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0" gracePeriod=600
Sep 06 10:24:34.864044 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:34.863845    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="config-reloader" containerID="cri-o://e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f" gracePeriod=600
Sep 06 10:24:34.864044 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:34.863862    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="thanos-sidecar" containerID="cri-o://c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a" gracePeriod=600
Sep 06 10:24:34.864044 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:34.863862    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898" gracePeriod=600
Sep 06 10:24:34.864044 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:34.863887    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy-web" containerID="cri-o://10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0" gracePeriod=600
Sep 06 10:24:34.864044 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:34.863818    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy" containerID="cri-o://17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2" gracePeriod=600
Sep 06 10:24:35.838877 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:35.838844    2564 generic.go:356] "Generic (PLEG): container finished" podID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerID="6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898" exitCode=0
Sep 06 10:24:35.838877 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:35.838868    2564 generic.go:356] "Generic (PLEG): container finished" podID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerID="17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2" exitCode=0
Sep 06 10:24:35.838877 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:35.838874    2564 generic.go:356] "Generic (PLEG): container finished" podID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerID="c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a" exitCode=0
Sep 06 10:24:35.838877 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:35.838880    2564 generic.go:356] "Generic (PLEG): container finished" podID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerID="e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f" exitCode=0
Sep 06 10:24:35.838877 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:35.838885    2564 generic.go:356] "Generic (PLEG): container finished" podID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerID="79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0" exitCode=0
Sep 06 10:24:35.839271 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:35.838917    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerDied","Data":"6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898"}
Sep 06 10:24:35.839271 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:35.839003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerDied","Data":"17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2"}
Sep 06 10:24:35.839271 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:35.839045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerDied","Data":"c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a"}
Sep 06 10:24:35.839271 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:35.839080    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerDied","Data":"e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f"}
Sep 06 10:24:35.839271 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:35.839118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerDied","Data":"79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0"}
Sep 06 10:24:36.106519 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.106496    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.190017 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.189974    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190037    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-trusted-ca-bundle\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190069    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-tls-assets\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190096    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-thanos-prometheus-http-client-file\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190153    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-kubelet-serving-ca-bundle\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190183    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-metrics-client-ca\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190235    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-config\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190266    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-0\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190291    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-config-out\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190323    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-kube-rbac-proxy\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190357    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-db\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190404    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-tls\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190433    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-web-config\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190456    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-1\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190491    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgmvv\" (UniqueName: \"kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-kube-api-access-dgmvv\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190944 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190525    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-serving-certs-ca-bundle\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190944 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190523    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:24:36.190944 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190554    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.190944 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.190599    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:24:36.191470 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.191189    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:24:36.191470 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.191398    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-2\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.191470 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.191442    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-metrics-client-certs\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.191680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.191476    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-grpc-tls\") pod \"a44e8a25-cbbf-4c45-b727-30bac91f7103\" (UID: \"a44e8a25-cbbf-4c45-b727-30bac91f7103\") "
Sep 06 10:24:36.191680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.191602    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:24:36.191782 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.191771    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.191833 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.191789    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.191833 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.191806    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.191833 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.191820    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-metrics-client-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.192049 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.192020    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:24:36.192103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.192075    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:24:36.192476 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.192449    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:24:36.195114 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.195092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:24:36.196052 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.196017    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-config-out" (OuterVolumeSpecName: "config-out") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:24:36.196135 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.196067    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:24:36.196135 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.196092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:24:36.196313 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.196260    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-config" (OuterVolumeSpecName: "config") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:24:36.196390 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.196352    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-kube-api-access-dgmvv" (OuterVolumeSpecName: "kube-api-access-dgmvv") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "kube-api-access-dgmvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:24:36.196575 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.196552    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:24:36.196817 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.196781    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:24:36.196888 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.196805    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:24:36.197061 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.197033    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:24:36.197155 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.197055    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:24:36.197155 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.197123    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:24:36.206036 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.206003    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-web-config" (OuterVolumeSpecName: "web-config") pod "a44e8a25-cbbf-4c45-b727-30bac91f7103" (UID: "a44e8a25-cbbf-4c45-b727-30bac91f7103"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:24:36.292593 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292552    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292593 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292592    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-tls-assets\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292607    2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292619    2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292629    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292638    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-config-out\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292647    2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-kube-rbac-proxy\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292656    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-db\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292664    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292672    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-web-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292680    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmvv\" (UniqueName: \"kubernetes.io/projected/a44e8a25-cbbf-4c45-b727-30bac91f7103-kube-api-access-dgmvv\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292688    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292698    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292707    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a44e8a25-cbbf-4c45-b727-30bac91f7103-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292717    2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-metrics-client-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.292787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.292725    2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a44e8a25-cbbf-4c45-b727-30bac91f7103-secret-grpc-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:24:36.849230 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.849170    2564 generic.go:356] "Generic (PLEG): container finished" podID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerID="10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0" exitCode=0
Sep 06 10:24:36.849521 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.849497    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerDied","Data":"10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0"}
Sep 06 10:24:36.849751 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.849736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a44e8a25-cbbf-4c45-b727-30bac91f7103","Type":"ContainerDied","Data":"1112d78cbda3cd7c77f826ce7fa20fffdb6cce27a5ba4964639fcc1dd0f037d7"}
Sep 06 10:24:36.849898 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.849885    2564 scope.go:117] "RemoveContainer" containerID="6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898"
Sep 06 10:24:36.850203 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.850179    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.860161 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.860137    2564 scope.go:117] "RemoveContainer" containerID="17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2"
Sep 06 10:24:36.868403 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.868385    2564 scope.go:117] "RemoveContainer" containerID="10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0"
Sep 06 10:24:36.875076 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.875057    2564 scope.go:117] "RemoveContainer" containerID="c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a"
Sep 06 10:24:36.879535 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.879511    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:24:36.883166 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.883144    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:24:36.883370 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.883357    2564 scope.go:117] "RemoveContainer" containerID="e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f"
Sep 06 10:24:36.890001 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.889984    2564 scope.go:117] "RemoveContainer" containerID="79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0"
Sep 06 10:24:36.897092 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.897069    2564 scope.go:117] "RemoveContainer" containerID="63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb"
Sep 06 10:24:36.903748 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.903733    2564 scope.go:117] "RemoveContainer" containerID="6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898"
Sep 06 10:24:36.904027 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:24:36.904002    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898\": container with ID starting with 6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898 not found: ID does not exist" containerID="6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898"
Sep 06 10:24:36.904074 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.904029    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898"} err="failed to get container status \"6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898\": rpc error: code = NotFound desc = could not find container \"6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898\": container with ID starting with 6bf39fb91b6f2d8413fef0cac171e09fa24b27773c17dd793fc56d2f5252c898 not found: ID does not exist"
Sep 06 10:24:36.904074 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.904061    2564 scope.go:117] "RemoveContainer" containerID="17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2"
Sep 06 10:24:36.904302 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:24:36.904282    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2\": container with ID starting with 17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2 not found: ID does not exist" containerID="17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2"
Sep 06 10:24:36.904390 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.904303    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2"} err="failed to get container status \"17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2\": rpc error: code = NotFound desc = could not find container \"17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2\": container with ID starting with 17c64a5b10580c95d44c0ecdb99d35c4dc00047368c678036f92bd558655faf2 not found: ID does not exist"
Sep 06 10:24:36.904390 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.904316    2564 scope.go:117] "RemoveContainer" containerID="10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0"
Sep 06 10:24:36.904555 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:24:36.904539    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0\": container with ID starting with 10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0 not found: ID does not exist" containerID="10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0"
Sep 06 10:24:36.904592 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.904569    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0"} err="failed to get container status \"10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0\": rpc error: code = NotFound desc = could not find container \"10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0\": container with ID starting with 10078c2302318b2afb0bae6b48c0b91b4b412f10ba13d88a83b8a73143309ba0 not found: ID does not exist"
Sep 06 10:24:36.904592 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.904585    2564 scope.go:117] "RemoveContainer" containerID="c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a"
Sep 06 10:24:36.904813 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:24:36.904798    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a\": container with ID starting with c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a not found: ID does not exist" containerID="c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a"
Sep 06 10:24:36.904854 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.904817    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a"} err="failed to get container status \"c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a\": rpc error: code = NotFound desc = could not find container \"c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a\": container with ID starting with c059453bcac8a57d8ed444632812dab45795a7d0d81f56422c80d8309b92df3a not found: ID does not exist"
Sep 06 10:24:36.904854 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.904830    2564 scope.go:117] "RemoveContainer" containerID="e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f"
Sep 06 10:24:36.905029 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:24:36.905016    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f\": container with ID starting with e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f not found: ID does not exist" containerID="e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f"
Sep 06 10:24:36.905063 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.905032    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f"} err="failed to get container status \"e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f\": rpc error: code = NotFound desc = could not find container \"e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f\": container with ID starting with e53a281f0dd316e433efad090dc047b8701f60214acf275410565c91f4c85c6f not found: ID does not exist"
Sep 06 10:24:36.905063 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.905044    2564 scope.go:117] "RemoveContainer" containerID="79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0"
Sep 06 10:24:36.905284 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:24:36.905258    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0\": container with ID starting with 79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0 not found: ID does not exist" containerID="79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0"
Sep 06 10:24:36.905284 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.905278    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0"} err="failed to get container status \"79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0\": rpc error: code = NotFound desc = could not find container \"79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0\": container with ID starting with 79007ca62d62b87570a76d2394a81d655d6b7f915b0df834f9a96556ac1bfea0 not found: ID does not exist"
Sep 06 10:24:36.905431 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.905292    2564 scope.go:117] "RemoveContainer" containerID="63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb"
Sep 06 10:24:36.905549 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:24:36.905534    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb\": container with ID starting with 63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb not found: ID does not exist" containerID="63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb"
Sep 06 10:24:36.905594 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.905552    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb"} err="failed to get container status \"63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb\": rpc error: code = NotFound desc = could not find container \"63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb\": container with ID starting with 63675feebb7c5a2abfd5e1cb2c87d5139a9401d96592c99a03c0505adad7d2fb not found: ID does not exist"
Sep 06 10:24:36.916277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916257    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:24:36.916566 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916552    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="config-reloader"
Sep 06 10:24:36.916605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916569    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="config-reloader"
Sep 06 10:24:36.916605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916586    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="prometheus"
Sep 06 10:24:36.916605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916594    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="prometheus"
Sep 06 10:24:36.916605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916602    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy"
Sep 06 10:24:36.916722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916607    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy"
Sep 06 10:24:36.916722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916613    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy-thanos"
Sep 06 10:24:36.916722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916618    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy-thanos"
Sep 06 10:24:36.916722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916627    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy-web"
Sep 06 10:24:36.916722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916633    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy-web"
Sep 06 10:24:36.916722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916640    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="thanos-sidecar"
Sep 06 10:24:36.916722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916645    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="thanos-sidecar"
Sep 06 10:24:36.916722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916658    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="init-config-reloader"
Sep 06 10:24:36.916722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916666    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="init-config-reloader"
Sep 06 10:24:36.916722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916725    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy-thanos"
Sep 06 10:24:36.916992 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916731    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="prometheus"
Sep 06 10:24:36.916992 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916737    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="thanos-sidecar"
Sep 06 10:24:36.916992 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916746    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy"
Sep 06 10:24:36.916992 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916753    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="kube-rbac-proxy-web"
Sep 06 10:24:36.916992 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.916759    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" containerName="config-reloader"
Sep 06 10:24:36.920523 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.920508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.923314 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.923290    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 06 10:24:36.923487 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.923468    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-71r72ghhmhfc4\""
Sep 06 10:24:36.923571 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.923526    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 06 10:24:36.923571 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.923473    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 06 10:24:36.923676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.923640    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 06 10:24:36.923731 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.923682    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 06 10:24:36.923820 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.923805    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 06 10:24:36.923820 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.923816    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 06 10:24:36.924019 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.924005    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 06 10:24:36.924090 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.924022    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4frx7\""
Sep 06 10:24:36.924143 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.924098    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 06 10:24:36.924459 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.924443    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 06 10:24:36.924527 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.924451    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 06 10:24:36.924905 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.924875    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 06 10:24:36.927845 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.927824    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 06 10:24:36.931086 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.931068    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 06 10:24:36.937925 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.937905    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:24:36.998425 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998403    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-config\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998550 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998550 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998452    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998550 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998470    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998550 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998716 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998556    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998716 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998716 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2dzj\" (UniqueName: \"kubernetes.io/projected/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-kube-api-access-b2dzj\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998716 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998620    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998716 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998716 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998716 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998692    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998718    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998762    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998782    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998795    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:36.998961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:36.998847    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100140 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100044    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100140 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100140 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100140 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100127    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100202    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-config\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100318    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100341    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2dzj\" (UniqueName: \"kubernetes.io/projected/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-kube-api-access-b2dzj\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.100574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.101297 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100616    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.101297 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.101297 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100892    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.101297 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100952    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.101297 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.100950    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.101643 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.101618    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.103638 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.102666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.103638 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.103367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.103638 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.103599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.103860 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.103652    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.103940 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.103916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.104008 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.103958    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.104135 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.104113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-config\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.104463 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.104439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.104809 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.104785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.105256 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.105236    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.105896 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.105869    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.106243 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.106203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.106336 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.106318    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.106478 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.106459    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.107077 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.107061    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.109193 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.109171    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2dzj\" (UniqueName: \"kubernetes.io/projected/ab6bb49a-fd68-4a39-8c03-83646f02a6f1-kube-api-access-b2dzj\") pod \"prometheus-k8s-0\" (UID: \"ab6bb49a-fd68-4a39-8c03-83646f02a6f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.231440 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.231404    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:24:37.516257 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.516233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:24:37.516948 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:24:37.516920    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6bb49a_fd68_4a39_8c03_83646f02a6f1.slice/crio-26bfd02dc82ce2d5e1ef826b2e65794c64b6069c0fb747d518d501c7ac7295f2 WatchSource:0}: Error finding container 26bfd02dc82ce2d5e1ef826b2e65794c64b6069c0fb747d518d501c7ac7295f2: Status 404 returned error can't find the container with id 26bfd02dc82ce2d5e1ef826b2e65794c64b6069c0fb747d518d501c7ac7295f2
Sep 06 10:24:37.855777 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.855745    2564 generic.go:356] "Generic (PLEG): container finished" podID="ab6bb49a-fd68-4a39-8c03-83646f02a6f1" containerID="fc117a01bda0c8cf904f7c8713745437205b36208d8a757cc2335d68d3544256" exitCode=0
Sep 06 10:24:37.855920 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.855831    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ab6bb49a-fd68-4a39-8c03-83646f02a6f1","Type":"ContainerDied","Data":"fc117a01bda0c8cf904f7c8713745437205b36208d8a757cc2335d68d3544256"}
Sep 06 10:24:37.855920 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:37.855897    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ab6bb49a-fd68-4a39-8c03-83646f02a6f1","Type":"ContainerStarted","Data":"26bfd02dc82ce2d5e1ef826b2e65794c64b6069c0fb747d518d501c7ac7295f2"}
Sep 06 10:24:38.028416 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:38.028378    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a44e8a25-cbbf-4c45-b727-30bac91f7103" path="/var/lib/kubelet/pods/a44e8a25-cbbf-4c45-b727-30bac91f7103/volumes"
Sep 06 10:24:38.865328 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:38.865294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ab6bb49a-fd68-4a39-8c03-83646f02a6f1","Type":"ContainerStarted","Data":"43e12f5dff1fcf3ad6ae5f9782eb47882f4498011538ec7e56deb915efeeb35b"}
Sep 06 10:24:38.865774 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:38.865390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ab6bb49a-fd68-4a39-8c03-83646f02a6f1","Type":"ContainerStarted","Data":"85ccc4a1b63b1af82c19f213a390b13390d5b94751696a410eeeccce3c763e98"}
Sep 06 10:24:38.865774 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:38.865417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ab6bb49a-fd68-4a39-8c03-83646f02a6f1","Type":"ContainerStarted","Data":"edb1ba097719ccca90b4153cfb6c6f88de8c5e432a84910a5d55d1477e34ffa7"}
Sep 06 10:24:38.865774 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:38.865442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ab6bb49a-fd68-4a39-8c03-83646f02a6f1","Type":"ContainerStarted","Data":"b7ca206983efc3d931a37ea8c7b8f066606db427d901a13927326b3a71f41d50"}
Sep 06 10:24:38.865774 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:38.865465    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ab6bb49a-fd68-4a39-8c03-83646f02a6f1","Type":"ContainerStarted","Data":"0c2342b91e9b08cbef10688305da5c4ae6958788e2d39fdfed322c28dff19354"}
Sep 06 10:24:38.865774 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:38.865487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ab6bb49a-fd68-4a39-8c03-83646f02a6f1","Type":"ContainerStarted","Data":"86ede82db1c21c6f9c380b0b06ad4b870303a61946928f83f513d101cc3fc936"}
Sep 06 10:24:38.898411 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:38.898347    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8792635 podStartE2EDuration="2.898326362s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.062862ms" isStatefulPod=false podCreationTimestamp="2026-09-06 10:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:24:38.896809514 +0000 UTC m=+167.380309642" watchObservedRunningTime="2026-09-06 10:24:38.898326362 +0000 UTC m=+167.381826472"
Sep 06 10:24:42.232080 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:24:42.232044    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:25:37.232094 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:25:37.232056    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:25:37.247666 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:25:37.247638    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:25:38.139348 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:25:38.139312    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:26:00.127597 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.127556    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"]
Sep 06 10:26:00.129984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.129964    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.140950 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.140929    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 06 10:26:00.140950 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.140946    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 06 10:26:00.142256 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.142241    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rsrp9\""
Sep 06 10:26:00.142422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.142410    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 06 10:26:00.142521 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.142501    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 06 10:26:00.143299 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.143283    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 06 10:26:00.154794 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.154765    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"]
Sep 06 10:26:00.205707 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.205670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52be845f-2e4a-489a-a926-82f33592a6ca-metrics-cert\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.205707 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.205706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52be845f-2e4a-489a-a926-82f33592a6ca-cert\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.205933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.205730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7nfq\" (UniqueName: \"kubernetes.io/projected/52be845f-2e4a-489a-a926-82f33592a6ca-kube-api-access-z7nfq\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.205933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.205801    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/52be845f-2e4a-489a-a926-82f33592a6ca-manager-config\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.306814 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.306777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/52be845f-2e4a-489a-a926-82f33592a6ca-manager-config\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.307005 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.306888    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52be845f-2e4a-489a-a926-82f33592a6ca-metrics-cert\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.307005 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.306922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52be845f-2e4a-489a-a926-82f33592a6ca-cert\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.307005 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.306945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7nfq\" (UniqueName: \"kubernetes.io/projected/52be845f-2e4a-489a-a926-82f33592a6ca-kube-api-access-z7nfq\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.307441 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.307410    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/52be845f-2e4a-489a-a926-82f33592a6ca-manager-config\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.309588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.309558    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52be845f-2e4a-489a-a926-82f33592a6ca-metrics-cert\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.309702 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.309633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52be845f-2e4a-489a-a926-82f33592a6ca-cert\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.319514 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.319491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7nfq\" (UniqueName: \"kubernetes.io/projected/52be845f-2e4a-489a-a926-82f33592a6ca-kube-api-access-z7nfq\") pod \"lws-controller-manager-5dc6f584c9-hz2fs\" (UID: \"52be845f-2e4a-489a-a926-82f33592a6ca\") " pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.438691 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.438608    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:00.720000 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:00.719970    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"]
Sep 06 10:26:00.722468 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:26:00.722444    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52be845f_2e4a_489a_a926_82f33592a6ca.slice/crio-cf76a98d4139d420f048e49fdf78fc3f9bc7e2f03e2cbae6f44580c46cd3fe8a WatchSource:0}: Error finding container cf76a98d4139d420f048e49fdf78fc3f9bc7e2f03e2cbae6f44580c46cd3fe8a: Status 404 returned error can't find the container with id cf76a98d4139d420f048e49fdf78fc3f9bc7e2f03e2cbae6f44580c46cd3fe8a
Sep 06 10:26:01.206329 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:01.206296    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs" event={"ID":"52be845f-2e4a-489a-a926-82f33592a6ca","Type":"ContainerStarted","Data":"cf76a98d4139d420f048e49fdf78fc3f9bc7e2f03e2cbae6f44580c46cd3fe8a"}
Sep 06 10:26:04.222105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:04.222068    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs" event={"ID":"52be845f-2e4a-489a-a926-82f33592a6ca","Type":"ContainerStarted","Data":"7144d153f5aaaca228d671cb54d0fa55376d4d909042bf5659fb43df5e83613c"}
Sep 06 10:26:04.222501 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:04.222226    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:04.238338 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:04.238295    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs" podStartSLOduration=1.52853347 podStartE2EDuration="4.238281118s" totalImagesPullingTime="2.709747648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:26:04.237159695 +0000 UTC m=+252.720659821" watchObservedRunningTime="2026-09-06 10:26:04.238281118 +0000 UTC m=+252.721781227"
Sep 06 10:26:08.467629 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.467592    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"]
Sep 06 10:26:08.470115 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.470095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.473036 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.473014    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 06 10:26:08.473036 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.473027    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-w5pz8\""
Sep 06 10:26:08.473225 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.473055    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 06 10:26:08.473225 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.473022    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 06 10:26:08.473505 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.473491    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 06 10:26:08.485797 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.485773    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"]
Sep 06 10:26:08.578375 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.578342    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/def6d513-734b-4979-b74e-ab5dbda86afe-manifests\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.578375 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.578376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/def6d513-734b-4979-b74e-ab5dbda86afe-tmp\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.578593 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.578438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/def6d513-734b-4979-b74e-ab5dbda86afe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.578593 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.578466    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/def6d513-734b-4979-b74e-ab5dbda86afe-webhook-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.578593 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.578501    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrmw\" (UniqueName: \"kubernetes.io/projected/def6d513-734b-4979-b74e-ab5dbda86afe-kube-api-access-5nrmw\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.679724 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.679685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/def6d513-734b-4979-b74e-ab5dbda86afe-manifests\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.679724 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.679724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/def6d513-734b-4979-b74e-ab5dbda86afe-tmp\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.679963 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.679766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/def6d513-734b-4979-b74e-ab5dbda86afe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.679963 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.679799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/def6d513-734b-4979-b74e-ab5dbda86afe-webhook-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.679963 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.679819    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrmw\" (UniqueName: \"kubernetes.io/projected/def6d513-734b-4979-b74e-ab5dbda86afe-kube-api-access-5nrmw\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.680153 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.680089    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/def6d513-734b-4979-b74e-ab5dbda86afe-manifests\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.682166 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.682140    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/def6d513-734b-4979-b74e-ab5dbda86afe-tmp\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.682292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.682224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/def6d513-734b-4979-b74e-ab5dbda86afe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.682377 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.682358    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/def6d513-734b-4979-b74e-ab5dbda86afe-webhook-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.689141 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.689120    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrmw\" (UniqueName: \"kubernetes.io/projected/def6d513-734b-4979-b74e-ab5dbda86afe-kube-api-access-5nrmw\") pod \"opendatahub-operator-controller-manager-c688cbfb6-g7qgl\" (UID: \"def6d513-734b-4979-b74e-ab5dbda86afe\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:08.780229 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:08.780118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:09.054591 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:09.054567    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"]
Sep 06 10:26:09.057481 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:26:09.057451    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef6d513_734b_4979_b74e_ab5dbda86afe.slice/crio-10d9494bfc490459f4ed908b4aee747c90e2fd5563a03eaef993bb3e9c624ca5 WatchSource:0}: Error finding container 10d9494bfc490459f4ed908b4aee747c90e2fd5563a03eaef993bb3e9c624ca5: Status 404 returned error can't find the container with id 10d9494bfc490459f4ed908b4aee747c90e2fd5563a03eaef993bb3e9c624ca5
Sep 06 10:26:09.242837 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:09.242804    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl" event={"ID":"def6d513-734b-4979-b74e-ab5dbda86afe","Type":"ContainerStarted","Data":"10d9494bfc490459f4ed908b4aee747c90e2fd5563a03eaef993bb3e9c624ca5"}
Sep 06 10:26:12.259528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:12.259447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl" event={"ID":"def6d513-734b-4979-b74e-ab5dbda86afe","Type":"ContainerStarted","Data":"99d4b28d05cf785ea98e185ac2a9fe061faeb05436df16dce2df10cd606bad08"}
Sep 06 10:26:13.265802 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:13.265718    2564 generic.go:356] "Generic (PLEG): container finished" podID="def6d513-734b-4979-b74e-ab5dbda86afe" containerID="99d4b28d05cf785ea98e185ac2a9fe061faeb05436df16dce2df10cd606bad08" exitCode=0
Sep 06 10:26:13.265802 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:13.265767    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl" event={"ID":"def6d513-734b-4979-b74e-ab5dbda86afe","Type":"ContainerDied","Data":"99d4b28d05cf785ea98e185ac2a9fe061faeb05436df16dce2df10cd606bad08"}
Sep 06 10:26:14.273000 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:14.272918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl" event={"ID":"def6d513-734b-4979-b74e-ab5dbda86afe","Type":"ContainerStarted","Data":"1222cd3d4864b0a2197beb561e936c08ff945f0d3ec1ab7a1982628d7d356ede"}
Sep 06 10:26:14.273386 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:14.273029    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:14.294975 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:14.294925    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl" podStartSLOduration=2.045441884 podStartE2EDuration="6.294909965s" totalImagesPullingTime="3.237363739s" totalInitContainerRuntime="1.012104342s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:26:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:26:14.29351827 +0000 UTC m=+262.777018396" watchObservedRunningTime="2026-09-06 10:26:14.294909965 +0000 UTC m=+262.778410073"
Sep 06 10:26:15.228751 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:15.228717    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5dc6f584c9-hz2fs"
Sep 06 10:26:25.279533 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:25.279497    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-g7qgl"
Sep 06 10:26:27.509918 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.509888    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"]
Sep 06 10:26:27.617407 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.617368    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"]
Sep 06 10:26:27.617601 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.617501    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:27.623137 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.623112    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 06 10:26:27.623272 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.623115    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 06 10:26:27.623585 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.623562    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2l794\""
Sep 06 10:26:27.623585 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.623579    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 06 10:26:27.623760 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.623565    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 06 10:26:27.737785 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.737755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba008a94-cc59-4b58-b4f8-7833f5cefc42-tmp\") pod \"kube-auth-proxy-68f5945d9f-m9pss\" (UID: \"ba008a94-cc59-4b58-b4f8-7833f5cefc42\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:27.737995 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.737816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba008a94-cc59-4b58-b4f8-7833f5cefc42-tls-certs\") pod \"kube-auth-proxy-68f5945d9f-m9pss\" (UID: \"ba008a94-cc59-4b58-b4f8-7833f5cefc42\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:27.737995 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.737916    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrb6h\" (UniqueName: \"kubernetes.io/projected/ba008a94-cc59-4b58-b4f8-7833f5cefc42-kube-api-access-qrb6h\") pod \"kube-auth-proxy-68f5945d9f-m9pss\" (UID: \"ba008a94-cc59-4b58-b4f8-7833f5cefc42\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:27.839245 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.839190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba008a94-cc59-4b58-b4f8-7833f5cefc42-tls-certs\") pod \"kube-auth-proxy-68f5945d9f-m9pss\" (UID: \"ba008a94-cc59-4b58-b4f8-7833f5cefc42\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:27.839421 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.839294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrb6h\" (UniqueName: \"kubernetes.io/projected/ba008a94-cc59-4b58-b4f8-7833f5cefc42-kube-api-access-qrb6h\") pod \"kube-auth-proxy-68f5945d9f-m9pss\" (UID: \"ba008a94-cc59-4b58-b4f8-7833f5cefc42\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:27.839421 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.839345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba008a94-cc59-4b58-b4f8-7833f5cefc42-tmp\") pod \"kube-auth-proxy-68f5945d9f-m9pss\" (UID: \"ba008a94-cc59-4b58-b4f8-7833f5cefc42\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:27.841617 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.841586    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba008a94-cc59-4b58-b4f8-7833f5cefc42-tmp\") pod \"kube-auth-proxy-68f5945d9f-m9pss\" (UID: \"ba008a94-cc59-4b58-b4f8-7833f5cefc42\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:27.841827 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.841809    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba008a94-cc59-4b58-b4f8-7833f5cefc42-tls-certs\") pod \"kube-auth-proxy-68f5945d9f-m9pss\" (UID: \"ba008a94-cc59-4b58-b4f8-7833f5cefc42\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:27.859300 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.859275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrb6h\" (UniqueName: \"kubernetes.io/projected/ba008a94-cc59-4b58-b4f8-7833f5cefc42-kube-api-access-qrb6h\") pod \"kube-auth-proxy-68f5945d9f-m9pss\" (UID: \"ba008a94-cc59-4b58-b4f8-7833f5cefc42\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:27.927379 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:27.927342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"
Sep 06 10:26:28.214847 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:28.214823    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss"]
Sep 06 10:26:28.217261 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:26:28.217230    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba008a94_cc59_4b58_b4f8_7833f5cefc42.slice/crio-c1804e519898a510693658f7aed14d64c91551f1619737da0743ce0d073a47a3 WatchSource:0}: Error finding container c1804e519898a510693658f7aed14d64c91551f1619737da0743ce0d073a47a3: Status 404 returned error can't find the container with id c1804e519898a510693658f7aed14d64c91551f1619737da0743ce0d073a47a3
Sep 06 10:26:28.342093 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:28.342055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss" event={"ID":"ba008a94-cc59-4b58-b4f8-7833f5cefc42","Type":"ContainerStarted","Data":"c1804e519898a510693658f7aed14d64c91551f1619737da0743ce0d073a47a3"}
Sep 06 10:26:31.358566 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:31.358529    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss" event={"ID":"ba008a94-cc59-4b58-b4f8-7833f5cefc42","Type":"ContainerStarted","Data":"5c59139b0b7d898689ad109f7df718cd76f30d2f5141653848b998479db0cabe"}
Sep 06 10:26:31.375035 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:31.374840    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68f5945d9f-m9pss" podStartSLOduration=1.783354121 podStartE2EDuration="4.374821277s" totalImagesPullingTime="2.591467156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:26:31.374099176 +0000 UTC m=+279.857599285" watchObservedRunningTime="2026-09-06 10:26:31.374821277 +0000 UTC m=+279.858321388"
Sep 06 10:26:51.909206 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:51.909174    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:26:51.911076 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:51.911052    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:26:51.914475 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:26:51.914453    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 06 10:28:06.401986 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.401954    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"]
Sep 06 10:28:06.405716 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.405698    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.408226 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.408189    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 06 10:28:06.408226 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.408201    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 06 10:28:06.408226 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.408204    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 06 10:28:06.409230 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.409190    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 06 10:28:06.409293 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.409193    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jx2cd\""
Sep 06 10:28:06.412707 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.412687    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"]
Sep 06 10:28:06.514560 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.514519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47sj\" (UniqueName: \"kubernetes.io/projected/0b321129-77f2-4396-9c55-22234b9f1847-kube-api-access-h47sj\") pod \"kuadrant-console-plugin-598f86468b-rpk2v\" (UID: \"0b321129-77f2-4396-9c55-22234b9f1847\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.514560 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.514559    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0b321129-77f2-4396-9c55-22234b9f1847-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rpk2v\" (UID: \"0b321129-77f2-4396-9c55-22234b9f1847\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.514789 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.514603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b321129-77f2-4396-9c55-22234b9f1847-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rpk2v\" (UID: \"0b321129-77f2-4396-9c55-22234b9f1847\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.615306 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.615273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47sj\" (UniqueName: \"kubernetes.io/projected/0b321129-77f2-4396-9c55-22234b9f1847-kube-api-access-h47sj\") pod \"kuadrant-console-plugin-598f86468b-rpk2v\" (UID: \"0b321129-77f2-4396-9c55-22234b9f1847\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.615306 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.615305    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0b321129-77f2-4396-9c55-22234b9f1847-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rpk2v\" (UID: \"0b321129-77f2-4396-9c55-22234b9f1847\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.615567 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.615341    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b321129-77f2-4396-9c55-22234b9f1847-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rpk2v\" (UID: \"0b321129-77f2-4396-9c55-22234b9f1847\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.615986 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.615961    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0b321129-77f2-4396-9c55-22234b9f1847-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rpk2v\" (UID: \"0b321129-77f2-4396-9c55-22234b9f1847\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.617993 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.617970    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b321129-77f2-4396-9c55-22234b9f1847-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rpk2v\" (UID: \"0b321129-77f2-4396-9c55-22234b9f1847\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.624468 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.624448    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47sj\" (UniqueName: \"kubernetes.io/projected/0b321129-77f2-4396-9c55-22234b9f1847-kube-api-access-h47sj\") pod \"kuadrant-console-plugin-598f86468b-rpk2v\" (UID: \"0b321129-77f2-4396-9c55-22234b9f1847\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.716081 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.715987    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"
Sep 06 10:28:06.989167 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.989130    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v"]
Sep 06 10:28:06.992012 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:28:06.991971    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b321129_77f2_4396_9c55_22234b9f1847.slice/crio-bbdc50341e71977f8c2cbc507a2c8264db0b06d134a0d9ac37de42a70e9e4f40 WatchSource:0}: Error finding container bbdc50341e71977f8c2cbc507a2c8264db0b06d134a0d9ac37de42a70e9e4f40: Status 404 returned error can't find the container with id bbdc50341e71977f8c2cbc507a2c8264db0b06d134a0d9ac37de42a70e9e4f40
Sep 06 10:28:06.993511 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:06.993491    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:28:07.779952 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:07.779919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v" event={"ID":"0b321129-77f2-4396-9c55-22234b9f1847","Type":"ContainerStarted","Data":"bbdc50341e71977f8c2cbc507a2c8264db0b06d134a0d9ac37de42a70e9e4f40"}
Sep 06 10:28:12.808435 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:12.808400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v" event={"ID":"0b321129-77f2-4396-9c55-22234b9f1847","Type":"ContainerStarted","Data":"04c7ed9b5b7378fb4cde85576d003713ab7ec765c316e7025ca6bb0adbf2ca62"}
Sep 06 10:28:12.826982 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:12.826928    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rpk2v" podStartSLOduration=2.056685169 podStartE2EDuration="6.826912756s" totalImagesPullingTime="4.770227587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:28:12.824734681 +0000 UTC m=+381.308234792" watchObservedRunningTime="2026-09-06 10:28:12.826912756 +0000 UTC m=+381.310412861"
Sep 06 10:28:16.471537 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.471500    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"]
Sep 06 10:28:16.474639 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.474624    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:16.477332 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.477307    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-frvzt\""
Sep 06 10:28:16.487680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.487660    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"]
Sep 06 10:28:16.585064 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.585029    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"]
Sep 06 10:28:16.585333 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:28:16.585312    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-zzckq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq" podUID="8ddc40b5-d571-4364-8b0a-cfc357d2c233"
Sep 06 10:28:16.609681 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.609652    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzckq\" (UniqueName: \"kubernetes.io/projected/8ddc40b5-d571-4364-8b0a-cfc357d2c233-kube-api-access-zzckq\") pod \"kuadrant-operator-controller-manager-6bd956ccf-gpsxq\" (UID: \"8ddc40b5-d571-4364-8b0a-cfc357d2c233\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:16.609822 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.609799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ddc40b5-d571-4364-8b0a-cfc357d2c233-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-gpsxq\" (UID: \"8ddc40b5-d571-4364-8b0a-cfc357d2c233\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:16.710984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.710940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzckq\" (UniqueName: \"kubernetes.io/projected/8ddc40b5-d571-4364-8b0a-cfc357d2c233-kube-api-access-zzckq\") pod \"kuadrant-operator-controller-manager-6bd956ccf-gpsxq\" (UID: \"8ddc40b5-d571-4364-8b0a-cfc357d2c233\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:16.711177 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.711068    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ddc40b5-d571-4364-8b0a-cfc357d2c233-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-gpsxq\" (UID: \"8ddc40b5-d571-4364-8b0a-cfc357d2c233\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:16.711464 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.711443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ddc40b5-d571-4364-8b0a-cfc357d2c233-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-gpsxq\" (UID: \"8ddc40b5-d571-4364-8b0a-cfc357d2c233\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:16.721615 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.721560    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzckq\" (UniqueName: \"kubernetes.io/projected/8ddc40b5-d571-4364-8b0a-cfc357d2c233-kube-api-access-zzckq\") pod \"kuadrant-operator-controller-manager-6bd956ccf-gpsxq\" (UID: \"8ddc40b5-d571-4364-8b0a-cfc357d2c233\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:16.826732 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.826705    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:16.833859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:16.833832    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:17.014313 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.014206    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ddc40b5-d571-4364-8b0a-cfc357d2c233-extensions-socket-volume\") pod \"8ddc40b5-d571-4364-8b0a-cfc357d2c233\" (UID: \"8ddc40b5-d571-4364-8b0a-cfc357d2c233\") "
Sep 06 10:28:17.014313 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.014302    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzckq\" (UniqueName: \"kubernetes.io/projected/8ddc40b5-d571-4364-8b0a-cfc357d2c233-kube-api-access-zzckq\") pod \"8ddc40b5-d571-4364-8b0a-cfc357d2c233\" (UID: \"8ddc40b5-d571-4364-8b0a-cfc357d2c233\") "
Sep 06 10:28:17.014494 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.014398    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddc40b5-d571-4364-8b0a-cfc357d2c233-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8ddc40b5-d571-4364-8b0a-cfc357d2c233" (UID: "8ddc40b5-d571-4364-8b0a-cfc357d2c233"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:28:17.014532 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.014499    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ddc40b5-d571-4364-8b0a-cfc357d2c233-extensions-socket-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:28:17.016255 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.016226    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddc40b5-d571-4364-8b0a-cfc357d2c233-kube-api-access-zzckq" (OuterVolumeSpecName: "kube-api-access-zzckq") pod "8ddc40b5-d571-4364-8b0a-cfc357d2c233" (UID: "8ddc40b5-d571-4364-8b0a-cfc357d2c233"). InnerVolumeSpecName "kube-api-access-zzckq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:28:17.115153 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.115119    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzckq\" (UniqueName: \"kubernetes.io/projected/8ddc40b5-d571-4364-8b0a-cfc357d2c233-kube-api-access-zzckq\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:28:17.186687 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.186655    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"]
Sep 06 10:28:17.195629 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.195604    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"]
Sep 06 10:28:17.251994 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.251964    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq"]
Sep 06 10:28:17.255561 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.255540    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq"
Sep 06 10:28:17.258302 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.258283    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5klng\""
Sep 06 10:28:17.269464 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.269383    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq"]
Sep 06 10:28:17.418462 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.418432    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dj9\" (UniqueName: \"kubernetes.io/projected/edf3481b-d9d7-48f8-8c51-3d6b4140abf3-kube-api-access-g5dj9\") pod \"limitador-operator-controller-manager-6c549b6d85-7xwwq\" (UID: \"edf3481b-d9d7-48f8-8c51-3d6b4140abf3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq"
Sep 06 10:28:17.519891 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.519804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dj9\" (UniqueName: \"kubernetes.io/projected/edf3481b-d9d7-48f8-8c51-3d6b4140abf3-kube-api-access-g5dj9\") pod \"limitador-operator-controller-manager-6c549b6d85-7xwwq\" (UID: \"edf3481b-d9d7-48f8-8c51-3d6b4140abf3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq"
Sep 06 10:28:17.528554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.528514    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dj9\" (UniqueName: \"kubernetes.io/projected/edf3481b-d9d7-48f8-8c51-3d6b4140abf3-kube-api-access-g5dj9\") pod \"limitador-operator-controller-manager-6c549b6d85-7xwwq\" (UID: \"edf3481b-d9d7-48f8-8c51-3d6b4140abf3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq"
Sep 06 10:28:17.566305 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.566271    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq"
Sep 06 10:28:17.831294 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.831266    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:17.833779 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:28:17.833747    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-gpsxq\" is forbidden: User \"system:node:ip-10-0-130-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-112.ec2.internal' and this object" podUID="8ddc40b5-d571-4364-8b0a-cfc357d2c233" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:17.839323 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:17.839303    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq"]
Sep 06 10:28:17.840480 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:28:17.840449    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-gpsxq\" is forbidden: User \"system:node:ip-10-0-130-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-112.ec2.internal' and this object" podUID="8ddc40b5-d571-4364-8b0a-cfc357d2c233" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-gpsxq"
Sep 06 10:28:17.841628 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:28:17.841604    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf3481b_d9d7_48f8_8c51_3d6b4140abf3.slice/crio-d16325e5ada26eedfd265a6af0457be000b93861bda44d5690ab1c29e83737ef WatchSource:0}: Error finding container d16325e5ada26eedfd265a6af0457be000b93861bda44d5690ab1c29e83737ef: Status 404 returned error can't find the container with id d16325e5ada26eedfd265a6af0457be000b93861bda44d5690ab1c29e83737ef
Sep 06 10:28:18.029342 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:18.029311    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ddc40b5-d571-4364-8b0a-cfc357d2c233" path="/var/lib/kubelet/pods/8ddc40b5-d571-4364-8b0a-cfc357d2c233/volumes"
Sep 06 10:28:18.839981 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:18.839879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq" event={"ID":"edf3481b-d9d7-48f8-8c51-3d6b4140abf3","Type":"ContainerStarted","Data":"d16325e5ada26eedfd265a6af0457be000b93861bda44d5690ab1c29e83737ef"}
Sep 06 10:28:20.851870 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:20.851833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq" event={"ID":"edf3481b-d9d7-48f8-8c51-3d6b4140abf3","Type":"ContainerStarted","Data":"9684090c7dea69740c61d7ef3b30c86d9f15b5016797957ac3db328325cf013f"}
Sep 06 10:28:20.852361 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:20.851933    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq"
Sep 06 10:28:20.870966 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:20.870924    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq" podStartSLOduration=1.231409037 podStartE2EDuration="3.870907791s" totalImagesPullingTime="2.639498754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:28:20.869114607 +0000 UTC m=+389.352614726" watchObservedRunningTime="2026-09-06 10:28:20.870907791 +0000 UTC m=+389.354407900"
Sep 06 10:28:31.858358 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:28:31.858322    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7xwwq"
Sep 06 10:29:03.476233 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:03.476182    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-dls96"]
Sep 06 10:29:03.479622 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:03.479599    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-dls96"
Sep 06 10:29:03.482549 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:03.482523    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-42x92\""
Sep 06 10:29:03.484058 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:03.484027    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-dls96"]
Sep 06 10:29:03.614048 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:03.614008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5jj9\" (UniqueName: \"kubernetes.io/projected/d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca-kube-api-access-d5jj9\") pod \"authorino-c548b5d7d-dls96\" (UID: \"d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca\") " pod="kuadrant-system/authorino-c548b5d7d-dls96"
Sep 06 10:29:03.714945 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:03.714887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5jj9\" (UniqueName: \"kubernetes.io/projected/d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca-kube-api-access-d5jj9\") pod \"authorino-c548b5d7d-dls96\" (UID: \"d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca\") " pod="kuadrant-system/authorino-c548b5d7d-dls96"
Sep 06 10:29:03.723593 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:03.723566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5jj9\" (UniqueName: \"kubernetes.io/projected/d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca-kube-api-access-d5jj9\") pod \"authorino-c548b5d7d-dls96\" (UID: \"d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca\") " pod="kuadrant-system/authorino-c548b5d7d-dls96"
Sep 06 10:29:03.791669 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:03.791629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-dls96"
Sep 06 10:29:04.060943 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:04.060782    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-dls96"]
Sep 06 10:29:04.063778 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:29:04.063750    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2a293ad_b3cd_4bc2_aa26_a1409e88c9ca.slice/crio-2e8289d7442f86e11b3fd9c27a4d71ac63913e2624e0abcb46b93222944fb43e WatchSource:0}: Error finding container 2e8289d7442f86e11b3fd9c27a4d71ac63913e2624e0abcb46b93222944fb43e: Status 404 returned error can't find the container with id 2e8289d7442f86e11b3fd9c27a4d71ac63913e2624e0abcb46b93222944fb43e
Sep 06 10:29:05.055843 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:05.055784    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-dls96" event={"ID":"d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca","Type":"ContainerStarted","Data":"2e8289d7442f86e11b3fd9c27a4d71ac63913e2624e0abcb46b93222944fb43e"}
Sep 06 10:29:06.061450 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:06.061415    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-dls96" event={"ID":"d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca","Type":"ContainerStarted","Data":"5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed"}
Sep 06 10:29:06.076785 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:06.076704    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-dls96" podStartSLOduration=1.297913369 podStartE2EDuration="3.076682498s" totalImagesPullingTime="1.778769129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:29:06.075856981 +0000 UTC m=+434.559357084" watchObservedRunningTime="2026-09-06 10:29:06.076682498 +0000 UTC m=+434.560182609"
Sep 06 10:29:34.430282 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.430241    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-dls96"]
Sep 06 10:29:34.430819 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.430462    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-dls96" podUID="d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca" containerName="authorino" containerID="cri-o://5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed" gracePeriod=30
Sep 06 10:29:34.515697 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.515669    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7498475b5d-2mgtf"]
Sep 06 10:29:34.519491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.519472    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498475b5d-2mgtf"
Sep 06 10:29:34.524243 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.522178    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 06 10:29:34.531192 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.531149    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7498475b5d-2mgtf"]
Sep 06 10:29:34.581187 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.581153    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8vbj\" (UniqueName: \"kubernetes.io/projected/ba0b6754-1c8a-4438-ab88-3e72a04f789b-kube-api-access-w8vbj\") pod \"authorino-7498475b5d-2mgtf\" (UID: \"ba0b6754-1c8a-4438-ab88-3e72a04f789b\") " pod="kuadrant-system/authorino-7498475b5d-2mgtf"
Sep 06 10:29:34.581387 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.581287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba0b6754-1c8a-4438-ab88-3e72a04f789b-tls-cert\") pod \"authorino-7498475b5d-2mgtf\" (UID: \"ba0b6754-1c8a-4438-ab88-3e72a04f789b\") " pod="kuadrant-system/authorino-7498475b5d-2mgtf"
Sep 06 10:29:34.677392 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.677367    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-dls96"
Sep 06 10:29:34.682670 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.682611    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8vbj\" (UniqueName: \"kubernetes.io/projected/ba0b6754-1c8a-4438-ab88-3e72a04f789b-kube-api-access-w8vbj\") pod \"authorino-7498475b5d-2mgtf\" (UID: \"ba0b6754-1c8a-4438-ab88-3e72a04f789b\") " pod="kuadrant-system/authorino-7498475b5d-2mgtf"
Sep 06 10:29:34.682670 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.682660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba0b6754-1c8a-4438-ab88-3e72a04f789b-tls-cert\") pod \"authorino-7498475b5d-2mgtf\" (UID: \"ba0b6754-1c8a-4438-ab88-3e72a04f789b\") " pod="kuadrant-system/authorino-7498475b5d-2mgtf"
Sep 06 10:29:34.684931 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.684913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba0b6754-1c8a-4438-ab88-3e72a04f789b-tls-cert\") pod \"authorino-7498475b5d-2mgtf\" (UID: \"ba0b6754-1c8a-4438-ab88-3e72a04f789b\") " pod="kuadrant-system/authorino-7498475b5d-2mgtf"
Sep 06 10:29:34.689655 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.689637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8vbj\" (UniqueName: \"kubernetes.io/projected/ba0b6754-1c8a-4438-ab88-3e72a04f789b-kube-api-access-w8vbj\") pod \"authorino-7498475b5d-2mgtf\" (UID: \"ba0b6754-1c8a-4438-ab88-3e72a04f789b\") " pod="kuadrant-system/authorino-7498475b5d-2mgtf"
Sep 06 10:29:34.783682 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.783646    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5jj9\" (UniqueName: \"kubernetes.io/projected/d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca-kube-api-access-d5jj9\") pod \"d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca\" (UID: \"d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca\") "
Sep 06 10:29:34.785806 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.785777    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca-kube-api-access-d5jj9" (OuterVolumeSpecName: "kube-api-access-d5jj9") pod "d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca" (UID: "d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca"). InnerVolumeSpecName "kube-api-access-d5jj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:29:34.834671 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.834645    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498475b5d-2mgtf"
Sep 06 10:29:34.884727 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:34.884694    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5jj9\" (UniqueName: \"kubernetes.io/projected/d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca-kube-api-access-d5jj9\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:29:35.113908 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.113885    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7498475b5d-2mgtf"]
Sep 06 10:29:35.115998 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:29:35.115968    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba0b6754_1c8a_4438_ab88_3e72a04f789b.slice/crio-abd6521ca4e265706b8169a6f2797cc700a20cd8961d5ee50035725f127f0d34 WatchSource:0}: Error finding container abd6521ca4e265706b8169a6f2797cc700a20cd8961d5ee50035725f127f0d34: Status 404 returned error can't find the container with id abd6521ca4e265706b8169a6f2797cc700a20cd8961d5ee50035725f127f0d34
Sep 06 10:29:35.198501 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.198467    2564 generic.go:356] "Generic (PLEG): container finished" podID="d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca" containerID="5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed" exitCode=0
Sep 06 10:29:35.198676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.198515    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-dls96"
Sep 06 10:29:35.198676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.198553    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-dls96" event={"ID":"d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca","Type":"ContainerDied","Data":"5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed"}
Sep 06 10:29:35.198676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.198592    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-dls96" event={"ID":"d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca","Type":"ContainerDied","Data":"2e8289d7442f86e11b3fd9c27a4d71ac63913e2624e0abcb46b93222944fb43e"}
Sep 06 10:29:35.198676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.198613    2564 scope.go:117] "RemoveContainer" containerID="5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed"
Sep 06 10:29:35.199825 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.199806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498475b5d-2mgtf" event={"ID":"ba0b6754-1c8a-4438-ab88-3e72a04f789b","Type":"ContainerStarted","Data":"abd6521ca4e265706b8169a6f2797cc700a20cd8961d5ee50035725f127f0d34"}
Sep 06 10:29:35.207481 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.207464    2564 scope.go:117] "RemoveContainer" containerID="5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed"
Sep 06 10:29:35.207727 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:29:35.207713    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed\": container with ID starting with 5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed not found: ID does not exist" containerID="5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed"
Sep 06 10:29:35.207767 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.207734    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed"} err="failed to get container status \"5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed\": rpc error: code = NotFound desc = could not find container \"5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed\": container with ID starting with 5c37762a25d6f7cb0fd180a753e304103b60f580b6b6e425125485a6efb850ed not found: ID does not exist"
Sep 06 10:29:35.220482 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.220455    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-dls96"]
Sep 06 10:29:35.223811 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:35.223791    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-dls96"]
Sep 06 10:29:36.029897 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:36.029861    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca" path="/var/lib/kubelet/pods/d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca/volumes"
Sep 06 10:29:36.205309 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:36.205264    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498475b5d-2mgtf" event={"ID":"ba0b6754-1c8a-4438-ab88-3e72a04f789b","Type":"ContainerStarted","Data":"449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32"}
Sep 06 10:29:36.222308 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:29:36.222256    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7498475b5d-2mgtf" podStartSLOduration=1.7864375749999999 podStartE2EDuration="2.222238725s" totalImagesPullingTime="435.80115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:29:36.22116988 +0000 UTC m=+464.704669991" watchObservedRunningTime="2026-09-06 10:29:36.222238725 +0000 UTC m=+464.705738825"
Sep 06 10:31:03.188185 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.188153    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"]
Sep 06 10:31:03.188775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.188762    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca" containerName="authorino"
Sep 06 10:31:03.188843 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.188783    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca" containerName="authorino"
Sep 06 10:31:03.188898 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.188891    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a293ad-b3cd-4bc2-aa26-a1409e88c9ca" containerName="authorino"
Sep 06 10:31:03.191717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.191696    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.194590 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.194568    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 06 10:31:03.194701 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.194610    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 06 10:31:03.195724 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.195706    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 06 10:31:03.195885 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.195867    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zm5kx\""
Sep 06 10:31:03.201798 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.201774    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"]
Sep 06 10:31:03.274383 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.274345    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47z2z\" (UniqueName: \"kubernetes.io/projected/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-kube-api-access-47z2z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.274383 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.274385    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.274588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.274411    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.274588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.274427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.274588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.274535    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.274588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.274565    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.375993 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.375944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.375993 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.375991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.376279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.376029    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47z2z\" (UniqueName: \"kubernetes.io/projected/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-kube-api-access-47z2z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.376279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.376049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.376279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.376073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.376279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.376090    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.376626 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.376594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.376702 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.376637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.376763 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.376705    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.378461 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.378440    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.378581 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.378562    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.383956 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.383926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47z2z\" (UniqueName: \"kubernetes.io/projected/e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3-kube-api-access-47z2z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bbsn\" (UID: \"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.502700 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.502600    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:03.777111 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:03.777084    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"]
Sep 06 10:31:03.779732 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:31:03.779700    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f5c2e8_d415_438b_bc6f_d7b7b506a3e3.slice/crio-e8c9b8136da85db35e2bf28359de2c07693f040409a484fff08573a598ce0ee3 WatchSource:0}: Error finding container e8c9b8136da85db35e2bf28359de2c07693f040409a484fff08573a598ce0ee3: Status 404 returned error can't find the container with id e8c9b8136da85db35e2bf28359de2c07693f040409a484fff08573a598ce0ee3
Sep 06 10:31:04.626471 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:04.626427    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn" event={"ID":"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3","Type":"ContainerStarted","Data":"e8c9b8136da85db35e2bf28359de2c07693f040409a484fff08573a598ce0ee3"}
Sep 06 10:31:05.632539 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:05.632502    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn" event={"ID":"e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3","Type":"ContainerStarted","Data":"5e5e8d0d92bd609a4b811450273d5b852dc2a075b7d7dbccb159ec789ee919d0"}
Sep 06 10:31:05.650833 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:05.650771    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn" podStartSLOduration=1.593823098 podStartE2EDuration="2.650753534s" totalImagesPullingTime="1.056930436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:05.649159142 +0000 UTC m=+554.132659264" watchObservedRunningTime="2026-09-06 10:31:05.650753534 +0000 UTC m=+554.134253643"
Sep 06 10:31:13.503305 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:13.503265    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:13.503766 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:13.503363    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:13.515778 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:13.515751    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:13.681256 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:13.681206    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bbsn"
Sep 06 10:31:29.593125 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.593089    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"]
Sep 06 10:31:29.596641 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.596620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.599313 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.599291    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 06 10:31:29.606406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.606385    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"]
Sep 06 10:31:29.618096 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.618074    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.618253 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.618108    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.618253 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.618138    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72503ff-78e2-4ef8-8cef-e88933f62c94-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.618345 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.618280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.618345 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.618320    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj6f6\" (UniqueName: \"kubernetes.io/projected/a72503ff-78e2-4ef8-8cef-e88933f62c94-kube-api-access-rj6f6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.618420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.618349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.719491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.719452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.719491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.719502    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.719770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.719535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72503ff-78e2-4ef8-8cef-e88933f62c94-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.719770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.719571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.719770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.719592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6f6\" (UniqueName: \"kubernetes.io/projected/a72503ff-78e2-4ef8-8cef-e88933f62c94-kube-api-access-rj6f6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.719770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.719611    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.719979 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.719915    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.720029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.719972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.720029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.719999    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.721845 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.721823    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72503ff-78e2-4ef8-8cef-e88933f62c94-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.722148 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.722129    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72503ff-78e2-4ef8-8cef-e88933f62c94-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.728043 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.728014    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj6f6\" (UniqueName: \"kubernetes.io/projected/a72503ff-78e2-4ef8-8cef-e88933f62c94-kube-api-access-rj6f6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6\" (UID: \"a72503ff-78e2-4ef8-8cef-e88933f62c94\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:29.906141 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:29.906054    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:30.179356 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:30.179260    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"]
Sep 06 10:31:30.182238 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:31:30.182179    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda72503ff_78e2_4ef8_8cef_e88933f62c94.slice/crio-cf8ffe268e8ba344e5dd5b975ca542f51ee5d4edc31738459a67cf6ce127118f WatchSource:0}: Error finding container cf8ffe268e8ba344e5dd5b975ca542f51ee5d4edc31738459a67cf6ce127118f: Status 404 returned error can't find the container with id cf8ffe268e8ba344e5dd5b975ca542f51ee5d4edc31738459a67cf6ce127118f
Sep 06 10:31:30.753977 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:30.753932    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6" event={"ID":"a72503ff-78e2-4ef8-8cef-e88933f62c94","Type":"ContainerStarted","Data":"a84db9fb742eb3d7f77e6cc543a7ce70c626ba5478494499f5a8736ce2ed6ed3"}
Sep 06 10:31:30.753977 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:30.753975    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6" event={"ID":"a72503ff-78e2-4ef8-8cef-e88933f62c94","Type":"ContainerStarted","Data":"cf8ffe268e8ba344e5dd5b975ca542f51ee5d4edc31738459a67cf6ce127118f"}
Sep 06 10:31:30.771411 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:30.771350    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6" podStartSLOduration=1.6138667930000001 podStartE2EDuration="1.771333801s" totalImagesPullingTime="157.467008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:30.770844686 +0000 UTC m=+579.254344795" watchObservedRunningTime="2026-09-06 10:31:30.771333801 +0000 UTC m=+579.254833910"
Sep 06 10:31:39.907207 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:39.907163    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:39.907822 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:39.907268    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:39.920136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:39.920110    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:40.814873 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:40.814838    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6"
Sep 06 10:31:51.945549 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:51.945516    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:31:51.946016 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:31:51.945825    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:32:31.624255 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:31.624221    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79b78b5455-djpqp"]
Sep 06 10:32:31.627550 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:31.627533    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b78b5455-djpqp"
Sep 06 10:32:31.636733 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:31.636710    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79b78b5455-djpqp"]
Sep 06 10:32:31.705707 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:31.705663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89c8d\" (UniqueName: \"kubernetes.io/projected/a9691cd0-c563-4bd8-8ecc-82bf98234a9d-kube-api-access-89c8d\") pod \"authorino-79b78b5455-djpqp\" (UID: \"a9691cd0-c563-4bd8-8ecc-82bf98234a9d\") " pod="kuadrant-system/authorino-79b78b5455-djpqp"
Sep 06 10:32:31.705888 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:31.705745    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9691cd0-c563-4bd8-8ecc-82bf98234a9d-tls-cert\") pod \"authorino-79b78b5455-djpqp\" (UID: \"a9691cd0-c563-4bd8-8ecc-82bf98234a9d\") " pod="kuadrant-system/authorino-79b78b5455-djpqp"
Sep 06 10:32:31.806996 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:31.806944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89c8d\" (UniqueName: \"kubernetes.io/projected/a9691cd0-c563-4bd8-8ecc-82bf98234a9d-kube-api-access-89c8d\") pod \"authorino-79b78b5455-djpqp\" (UID: \"a9691cd0-c563-4bd8-8ecc-82bf98234a9d\") " pod="kuadrant-system/authorino-79b78b5455-djpqp"
Sep 06 10:32:31.807135 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:31.807087    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9691cd0-c563-4bd8-8ecc-82bf98234a9d-tls-cert\") pod \"authorino-79b78b5455-djpqp\" (UID: \"a9691cd0-c563-4bd8-8ecc-82bf98234a9d\") " pod="kuadrant-system/authorino-79b78b5455-djpqp"
Sep 06 10:32:31.809558 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:31.809527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9691cd0-c563-4bd8-8ecc-82bf98234a9d-tls-cert\") pod \"authorino-79b78b5455-djpqp\" (UID: \"a9691cd0-c563-4bd8-8ecc-82bf98234a9d\") " pod="kuadrant-system/authorino-79b78b5455-djpqp"
Sep 06 10:32:31.815098 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:31.815071    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89c8d\" (UniqueName: \"kubernetes.io/projected/a9691cd0-c563-4bd8-8ecc-82bf98234a9d-kube-api-access-89c8d\") pod \"authorino-79b78b5455-djpqp\" (UID: \"a9691cd0-c563-4bd8-8ecc-82bf98234a9d\") " pod="kuadrant-system/authorino-79b78b5455-djpqp"
Sep 06 10:32:31.937444 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:31.937357    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b78b5455-djpqp"
Sep 06 10:32:32.210130 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:32.210071    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79b78b5455-djpqp"]
Sep 06 10:32:32.212382 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:32:32.212343    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9691cd0_c563_4bd8_8ecc_82bf98234a9d.slice/crio-501df9b9e4a4b979292ca756d3bcea745c1c009bb1f03750079f505011f04a7d WatchSource:0}: Error finding container 501df9b9e4a4b979292ca756d3bcea745c1c009bb1f03750079f505011f04a7d: Status 404 returned error can't find the container with id 501df9b9e4a4b979292ca756d3bcea745c1c009bb1f03750079f505011f04a7d
Sep 06 10:32:33.064853 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.064810    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79b78b5455-djpqp" event={"ID":"a9691cd0-c563-4bd8-8ecc-82bf98234a9d","Type":"ContainerStarted","Data":"918892805dd43a29489217ea708ec4f0680f8a36c6ce6ca4c8e57064cefe8fcd"}
Sep 06 10:32:33.064853 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.064857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79b78b5455-djpqp" event={"ID":"a9691cd0-c563-4bd8-8ecc-82bf98234a9d","Type":"ContainerStarted","Data":"501df9b9e4a4b979292ca756d3bcea745c1c009bb1f03750079f505011f04a7d"}
Sep 06 10:32:33.082598 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.082522    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79b78b5455-djpqp" podStartSLOduration=1.537740174 podStartE2EDuration="2.082496375s" totalImagesPullingTime="544.756201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:32:33.080084397 +0000 UTC m=+641.563584507" watchObservedRunningTime="2026-09-06 10:32:33.082496375 +0000 UTC m=+641.565996537"
Sep 06 10:32:33.112261 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.112221    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7498475b5d-2mgtf"]
Sep 06 10:32:33.112520 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.112490    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7498475b5d-2mgtf" podUID="ba0b6754-1c8a-4438-ab88-3e72a04f789b" containerName="authorino" containerID="cri-o://449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32" gracePeriod=30
Sep 06 10:32:33.375856 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.375831    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498475b5d-2mgtf"
Sep 06 10:32:33.523877 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.523841    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba0b6754-1c8a-4438-ab88-3e72a04f789b-tls-cert\") pod \"ba0b6754-1c8a-4438-ab88-3e72a04f789b\" (UID: \"ba0b6754-1c8a-4438-ab88-3e72a04f789b\") "
Sep 06 10:32:33.524066 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.523889    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8vbj\" (UniqueName: \"kubernetes.io/projected/ba0b6754-1c8a-4438-ab88-3e72a04f789b-kube-api-access-w8vbj\") pod \"ba0b6754-1c8a-4438-ab88-3e72a04f789b\" (UID: \"ba0b6754-1c8a-4438-ab88-3e72a04f789b\") "
Sep 06 10:32:33.525899 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.525860    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0b6754-1c8a-4438-ab88-3e72a04f789b-kube-api-access-w8vbj" (OuterVolumeSpecName: "kube-api-access-w8vbj") pod "ba0b6754-1c8a-4438-ab88-3e72a04f789b" (UID: "ba0b6754-1c8a-4438-ab88-3e72a04f789b"). InnerVolumeSpecName "kube-api-access-w8vbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:32:33.535743 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.535714    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0b6754-1c8a-4438-ab88-3e72a04f789b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ba0b6754-1c8a-4438-ab88-3e72a04f789b" (UID: "ba0b6754-1c8a-4438-ab88-3e72a04f789b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:32:33.625386 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.625290    2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba0b6754-1c8a-4438-ab88-3e72a04f789b-tls-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:32:33.625386 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:33.625334    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8vbj\" (UniqueName: \"kubernetes.io/projected/ba0b6754-1c8a-4438-ab88-3e72a04f789b-kube-api-access-w8vbj\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:32:34.071607 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:34.071571    2564 generic.go:356] "Generic (PLEG): container finished" podID="ba0b6754-1c8a-4438-ab88-3e72a04f789b" containerID="449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32" exitCode=0
Sep 06 10:32:34.072086 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:34.071632    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498475b5d-2mgtf"
Sep 06 10:32:34.072086 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:34.071661    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498475b5d-2mgtf" event={"ID":"ba0b6754-1c8a-4438-ab88-3e72a04f789b","Type":"ContainerDied","Data":"449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32"}
Sep 06 10:32:34.072086 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:34.071720    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498475b5d-2mgtf" event={"ID":"ba0b6754-1c8a-4438-ab88-3e72a04f789b","Type":"ContainerDied","Data":"abd6521ca4e265706b8169a6f2797cc700a20cd8961d5ee50035725f127f0d34"}
Sep 06 10:32:34.072086 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:34.071745    2564 scope.go:117] "RemoveContainer" containerID="449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32"
Sep 06 10:32:34.083541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:34.082699    2564 scope.go:117] "RemoveContainer" containerID="449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32"
Sep 06 10:32:34.083541 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:32:34.083198    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32\": container with ID starting with 449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32 not found: ID does not exist" containerID="449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32"
Sep 06 10:32:34.083541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:34.083252    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32"} err="failed to get container status \"449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32\": rpc error: code = NotFound desc = could not find container \"449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32\": container with ID starting with 449a4dba9ff87a8a15d626684d0b4adc36bff5ee5b150bd27ad003b20c52ef32 not found: ID does not exist"
Sep 06 10:32:34.100739 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:34.100700    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7498475b5d-2mgtf"]
Sep 06 10:32:34.114045 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:34.114007    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7498475b5d-2mgtf"]
Sep 06 10:32:36.029257 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:36.029202    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba0b6754-1c8a-4438-ab88-3e72a04f789b" path="/var/lib/kubelet/pods/ba0b6754-1c8a-4438-ab88-3e72a04f789b/volumes"
Sep 06 10:32:52.065554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.065520    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7849-139883902498096"]
Sep 06 10:32:52.065926 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.065887    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba0b6754-1c8a-4438-ab88-3e72a04f789b" containerName="authorino"
Sep 06 10:32:52.065926 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.065898    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba0b6754-1c8a-4438-ab88-3e72a04f789b" containerName="authorino"
Sep 06 10:32:52.065999 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.065954    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba0b6754-1c8a-4438-ab88-3e72a04f789b" containerName="authorino"
Sep 06 10:32:52.070103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.070084    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:52.073055 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.073026    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-728g6\""
Sep 06 10:32:52.075488 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.075466    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-139883902498096"]
Sep 06 10:32:52.087494 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.087456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqg54\" (UniqueName: \"kubernetes.io/projected/5a8047e0-df08-402b-8a1b-4b575a742f8d-kube-api-access-mqg54\") pod \"test-curl-7849-139883902498096\" (UID: \"5a8047e0-df08-402b-8a1b-4b575a742f8d\") " pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:52.188234 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.188174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqg54\" (UniqueName: \"kubernetes.io/projected/5a8047e0-df08-402b-8a1b-4b575a742f8d-kube-api-access-mqg54\") pod \"test-curl-7849-139883902498096\" (UID: \"5a8047e0-df08-402b-8a1b-4b575a742f8d\") " pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:52.196326 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.196304    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqg54\" (UniqueName: \"kubernetes.io/projected/5a8047e0-df08-402b-8a1b-4b575a742f8d-kube-api-access-mqg54\") pod \"test-curl-7849-139883902498096\" (UID: \"5a8047e0-df08-402b-8a1b-4b575a742f8d\") " pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:52.380383 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.380348    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:52.650030 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:52.650005    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-139883902498096"]
Sep 06 10:32:52.651970 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:32:52.651934    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a8047e0_df08_402b_8a1b_4b575a742f8d.slice/crio-c63bc9199bc6b621fc9f7f77c41eaa4298dcedcf5b993cf47cb7bc35c83d9f74 WatchSource:0}: Error finding container c63bc9199bc6b621fc9f7f77c41eaa4298dcedcf5b993cf47cb7bc35c83d9f74: Status 404 returned error can't find the container with id c63bc9199bc6b621fc9f7f77c41eaa4298dcedcf5b993cf47cb7bc35c83d9f74
Sep 06 10:32:53.171415 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:53.171376    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-139883902498096" event={"ID":"5a8047e0-df08-402b-8a1b-4b575a742f8d","Type":"ContainerStarted","Data":"c63bc9199bc6b621fc9f7f77c41eaa4298dcedcf5b993cf47cb7bc35c83d9f74"}
Sep 06 10:32:55.183157 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:55.183122    2564 generic.go:356] "Generic (PLEG): container finished" podID="5a8047e0-df08-402b-8a1b-4b575a742f8d" containerID="494f1215c68ff0554ce70c3c42759399493a1e92760717894e6e9f97020b2555" exitCode=0
Sep 06 10:32:55.183555 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:55.183206    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-139883902498096" event={"ID":"5a8047e0-df08-402b-8a1b-4b575a742f8d","Type":"ContainerDied","Data":"494f1215c68ff0554ce70c3c42759399493a1e92760717894e6e9f97020b2555"}
Sep 06 10:32:56.316201 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.316174    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:56.383256 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.383203    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7849-139883902498096_5a8047e0-df08-402b-8a1b-4b575a742f8d/test-curl-7849-139883902498096/0.log"
Sep 06 10:32:56.405747 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.405717    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7849-139883902498096"]
Sep 06 10:32:56.409231 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.409194    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7849-139883902498096"]
Sep 06 10:32:56.424870 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.424848    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqg54\" (UniqueName: \"kubernetes.io/projected/5a8047e0-df08-402b-8a1b-4b575a742f8d-kube-api-access-mqg54\") pod \"5a8047e0-df08-402b-8a1b-4b575a742f8d\" (UID: \"5a8047e0-df08-402b-8a1b-4b575a742f8d\") "
Sep 06 10:32:56.427072 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.427047    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8047e0-df08-402b-8a1b-4b575a742f8d-kube-api-access-mqg54" (OuterVolumeSpecName: "kube-api-access-mqg54") pod "5a8047e0-df08-402b-8a1b-4b575a742f8d" (UID: "5a8047e0-df08-402b-8a1b-4b575a742f8d"). InnerVolumeSpecName "kube-api-access-mqg54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:32:56.525885 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.525796    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqg54\" (UniqueName: \"kubernetes.io/projected/5a8047e0-df08-402b-8a1b-4b575a742f8d-kube-api-access-mqg54\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:32:56.527487 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.527462    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7849-139883902498096"]
Sep 06 10:32:56.527839 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.527827    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8047e0-df08-402b-8a1b-4b575a742f8d" containerName="test-curl-7849-139883902498096"
Sep 06 10:32:56.527879 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.527841    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8047e0-df08-402b-8a1b-4b575a742f8d" containerName="test-curl-7849-139883902498096"
Sep 06 10:32:56.527910 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.527901    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8047e0-df08-402b-8a1b-4b575a742f8d" containerName="test-curl-7849-139883902498096"
Sep 06 10:32:56.531020 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.531002    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:56.537540 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.537475    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-139883902498096"]
Sep 06 10:32:56.627175 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.627130    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z59zq\" (UniqueName: \"kubernetes.io/projected/20aa6682-e171-4c42-95ff-b29eded5b761-kube-api-access-z59zq\") pod \"test-curl-7849-139883902498096\" (UID: \"20aa6682-e171-4c42-95ff-b29eded5b761\") " pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:56.727962 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.727922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z59zq\" (UniqueName: \"kubernetes.io/projected/20aa6682-e171-4c42-95ff-b29eded5b761-kube-api-access-z59zq\") pod \"test-curl-7849-139883902498096\" (UID: \"20aa6682-e171-4c42-95ff-b29eded5b761\") " pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:56.736167 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.736139    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z59zq\" (UniqueName: \"kubernetes.io/projected/20aa6682-e171-4c42-95ff-b29eded5b761-kube-api-access-z59zq\") pod \"test-curl-7849-139883902498096\" (UID: \"20aa6682-e171-4c42-95ff-b29eded5b761\") " pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:56.841521 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:56.841485    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:57.109422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:57.109391    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-139883902498096"]
Sep 06 10:32:57.111822 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:32:57.111797    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20aa6682_e171_4c42_95ff_b29eded5b761.slice/crio-ce354f9da6a3022553763de91ed6628e9c26898f91bbb0a5909385e34437757e WatchSource:0}: Error finding container ce354f9da6a3022553763de91ed6628e9c26898f91bbb0a5909385e34437757e: Status 404 returned error can't find the container with id ce354f9da6a3022553763de91ed6628e9c26898f91bbb0a5909385e34437757e
Sep 06 10:32:57.194676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:57.194649    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:57.194852 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:57.194697    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c63bc9199bc6b621fc9f7f77c41eaa4298dcedcf5b993cf47cb7bc35c83d9f74"
Sep 06 10:32:57.195947 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:57.195923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-139883902498096" event={"ID":"20aa6682-e171-4c42-95ff-b29eded5b761","Type":"ContainerStarted","Data":"ce354f9da6a3022553763de91ed6628e9c26898f91bbb0a5909385e34437757e"}
Sep 06 10:32:57.207076 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:57.207045    2564 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7849-139883902498096" oldPodUID="5a8047e0-df08-402b-8a1b-4b575a742f8d" podUID="20aa6682-e171-4c42-95ff-b29eded5b761"
Sep 06 10:32:58.029699 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:58.029661    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8047e0-df08-402b-8a1b-4b575a742f8d" path="/var/lib/kubelet/pods/5a8047e0-df08-402b-8a1b-4b575a742f8d/volumes"
Sep 06 10:32:58.201659 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:58.201622    2564 generic.go:356] "Generic (PLEG): container finished" podID="20aa6682-e171-4c42-95ff-b29eded5b761" containerID="67f4d2bfd044d8e386c2b4ed1451b1f0fe043a7c4bd02d093ffd3c02633dbded" exitCode=0
Sep 06 10:32:58.201822 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:58.201709    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-139883902498096" event={"ID":"20aa6682-e171-4c42-95ff-b29eded5b761","Type":"ContainerDied","Data":"67f4d2bfd044d8e386c2b4ed1451b1f0fe043a7c4bd02d093ffd3c02633dbded"}
Sep 06 10:32:59.346995 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:59.346958    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:32:59.366629 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:59.366596    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7849-139883902498096_20aa6682-e171-4c42-95ff-b29eded5b761/test-curl-7849-139883902498096/0.log"
Sep 06 10:32:59.386907 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:59.386870    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7849-139883902498096"]
Sep 06 10:32:59.391871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:59.391841    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7849-139883902498096"]
Sep 06 10:32:59.451737 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:59.451705    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z59zq\" (UniqueName: \"kubernetes.io/projected/20aa6682-e171-4c42-95ff-b29eded5b761-kube-api-access-z59zq\") pod \"20aa6682-e171-4c42-95ff-b29eded5b761\" (UID: \"20aa6682-e171-4c42-95ff-b29eded5b761\") "
Sep 06 10:32:59.453714 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:59.453693    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20aa6682-e171-4c42-95ff-b29eded5b761-kube-api-access-z59zq" (OuterVolumeSpecName: "kube-api-access-z59zq") pod "20aa6682-e171-4c42-95ff-b29eded5b761" (UID: "20aa6682-e171-4c42-95ff-b29eded5b761"). InnerVolumeSpecName "kube-api-access-z59zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:32:59.552875 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:32:59.552833    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z59zq\" (UniqueName: \"kubernetes.io/projected/20aa6682-e171-4c42-95ff-b29eded5b761-kube-api-access-z59zq\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:33:00.028558 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.028529    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20aa6682-e171-4c42-95ff-b29eded5b761" path="/var/lib/kubelet/pods/20aa6682-e171-4c42-95ff-b29eded5b761/volumes"
Sep 06 10:33:00.171016 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.170982    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"]
Sep 06 10:33:00.171404 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.171388    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20aa6682-e171-4c42-95ff-b29eded5b761" containerName="test-curl-7849-139883902498096"
Sep 06 10:33:00.171476 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.171407    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="20aa6682-e171-4c42-95ff-b29eded5b761" containerName="test-curl-7849-139883902498096"
Sep 06 10:33:00.171528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.171491    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="20aa6682-e171-4c42-95ff-b29eded5b761" containerName="test-curl-7849-139883902498096"
Sep 06 10:33:00.174358 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.174340    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.177290 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.177269    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-5dda3d-openshift-default-dockercfg-bpwfc\""
Sep 06 10:33:00.177416 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.177274    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 06 10:33:00.185831 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.185797    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"]
Sep 06 10:33:00.213989 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.213963    2564 scope.go:117] "RemoveContainer" containerID="67f4d2bfd044d8e386c2b4ed1451b1f0fe043a7c4bd02d093ffd3c02633dbded"
Sep 06 10:33:00.213989 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.213979    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-139883902498096"
Sep 06 10:33:00.259355 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.259312    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istiod-ca-cert\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.259524 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.259364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfcbj\" (UniqueName: \"kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-kube-api-access-nfcbj\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.259524 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.259413    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-data\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.259524 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.259452    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-podinfo\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.259641 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.259548    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-credential-socket\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.259641 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.259575    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-token\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.259641 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.259593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-envoy\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.259641 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.259640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-certs\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.259779 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.259686    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-socket\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.360968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istiod-ca-cert\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361011    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfcbj\" (UniqueName: \"kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-kube-api-access-nfcbj\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-data\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-podinfo\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-credential-socket\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-token\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361142    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-envoy\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-certs\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361286    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-socket\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361538    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-data\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361940 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361695    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-socket\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361940 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361759    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-credential-socket\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361940 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361819    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istiod-ca-cert\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.361940 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.361850    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-certs\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.363552 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.363529    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-envoy\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.363755 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.363739    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-podinfo\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.376234 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.376186    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-token\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.376356 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.376308    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfcbj\" (UniqueName: \"kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-kube-api-access-nfcbj\") pod \"e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") " pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.485413 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.485368    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:00.777888 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:00.777855    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"]
Sep 06 10:33:00.779365 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:33:00.779337    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4104a875_ff2b_432e_af78_1ed71b7fdd7b.slice/crio-5e7b45dca0312ab2c3876d35591408b932e65752de2787c33510e34fffe07c2a WatchSource:0}: Error finding container 5e7b45dca0312ab2c3876d35591408b932e65752de2787c33510e34fffe07c2a: Status 404 returned error can't find the container with id 5e7b45dca0312ab2c3876d35591408b932e65752de2787c33510e34fffe07c2a
Sep 06 10:33:01.221533 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:01.221498    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" event={"ID":"4104a875-ff2b-432e-af78-1ed71b7fdd7b","Type":"ContainerStarted","Data":"5e7b45dca0312ab2c3876d35591408b932e65752de2787c33510e34fffe07c2a"}
Sep 06 10:33:02.100027 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.099724    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"]
Sep 06 10:33:02.104388 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.104237    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.107137 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.107112    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 10:33:02.109473 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.108518    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-965kn\""
Sep 06 10:33:02.109473 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.108776    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 10:33:02.109473 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.108988    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 10:33:02.109473 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.109188    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dbaf75b1-serving-cert\""
Sep 06 10:33:02.118265 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.115906    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"]
Sep 06 10:33:02.184617 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.184579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-metrics-certs\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.184815 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.184647    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd5h8\" (UniqueName: \"kubernetes.io/projected/b86e1700-1ee3-460e-969c-9593f94bead3-kube-api-access-bd5h8\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.184815 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.184736    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86e1700-1ee3-460e-969c-9593f94bead3-openshift-service-ca\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.184815 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.184795    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.286306 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.286269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86e1700-1ee3-460e-969c-9593f94bead3-openshift-service-ca\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.286517 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.286449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.286517 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.286507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-metrics-certs\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.286634 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.286565    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd5h8\" (UniqueName: \"kubernetes.io/projected/b86e1700-1ee3-460e-969c-9593f94bead3-kube-api-access-bd5h8\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.286634 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:33:02.286598    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-serving-cert: secret "maas-api-e2e-ait-dbaf75b1-serving-cert" not found
Sep 06 10:33:02.286732 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:33:02.286674    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls podName:b86e1700-1ee3-460e-969c-9593f94bead3 nodeName:}" failed. No retries permitted until 2026-09-06 10:33:02.786646993 +0000 UTC m=+671.270147096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls") pod "maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6" (UID: "b86e1700-1ee3-460e-969c-9593f94bead3") : secret "maas-api-e2e-ait-dbaf75b1-serving-cert" not found
Sep 06 10:33:02.287142 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.287113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86e1700-1ee3-460e-969c-9593f94bead3-openshift-service-ca\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.289890 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.289867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-metrics-certs\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.295709 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.295615    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd5h8\" (UniqueName: \"kubernetes.io/projected/b86e1700-1ee3-460e-969c-9593f94bead3-kube-api-access-bd5h8\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.791966 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:02.791915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:02.792169 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:33:02.792102    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-serving-cert: secret "maas-api-e2e-ait-dbaf75b1-serving-cert" not found
Sep 06 10:33:02.792260 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:33:02.792178    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls podName:b86e1700-1ee3-460e-969c-9593f94bead3 nodeName:}" failed. No retries permitted until 2026-09-06 10:33:03.792156006 +0000 UTC m=+672.275656102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls") pod "maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6" (UID: "b86e1700-1ee3-460e-969c-9593f94bead3") : secret "maas-api-e2e-ait-dbaf75b1-serving-cert" not found
Sep 06 10:33:03.488984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:03.488944    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:33:03.489278 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:03.489023    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:33:03.489278 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:03.489053    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:33:03.802087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:03.802033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:03.804501 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:03.804470    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls\") pod \"maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:03.922286 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:03.922243    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:04.197339 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.197313    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"]
Sep 06 10:33:04.199852 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:33:04.199825    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86e1700_1ee3_460e_969c_9593f94bead3.slice/crio-7bc90a4922e6e32a636f54e2139dca4d8857f9afe1c7387cf9654ecb16c5bc2e WatchSource:0}: Error finding container 7bc90a4922e6e32a636f54e2139dca4d8857f9afe1c7387cf9654ecb16c5bc2e: Status 404 returned error can't find the container with id 7bc90a4922e6e32a636f54e2139dca4d8857f9afe1c7387cf9654ecb16c5bc2e
Sep 06 10:33:04.238509 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.238469    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6" event={"ID":"b86e1700-1ee3-460e-969c-9593f94bead3","Type":"ContainerStarted","Data":"7bc90a4922e6e32a636f54e2139dca4d8857f9afe1c7387cf9654ecb16c5bc2e"}
Sep 06 10:33:04.240125 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.240092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" event={"ID":"4104a875-ff2b-432e-af78-1ed71b7fdd7b","Type":"ContainerStarted","Data":"6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318"}
Sep 06 10:33:04.259778 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.259721    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" podStartSLOduration=1.552647018 podStartE2EDuration="4.259703734s" totalImagesPullingTime="2.707056716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:04.25876158 +0000 UTC m=+672.742261712" watchObservedRunningTime="2026-09-06 10:33:04.259703734 +0000 UTC m=+672.743203843"
Sep 06 10:33:04.307946 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.307910    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"]
Sep 06 10:33:04.312491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.312468    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.314823 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.314803    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-30549304-kserve-self-signed-certs\""
Sep 06 10:33:04.319607 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.319583    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"]
Sep 06 10:33:04.407400 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.407353    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swhbb\" (UniqueName: \"kubernetes.io/projected/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-kube-api-access-swhbb\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.407400 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.407400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-dshm\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.407591 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.407425    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tls-certs\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.407591 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.407526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-home\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.407591 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.407561    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tmp-dir\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.407712 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.407610    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-model-cache\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.486646 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.486553    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:04.491564 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.491539    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:04.508942 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.508911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-model-cache\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.509051 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.508968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swhbb\" (UniqueName: \"kubernetes.io/projected/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-kube-api-access-swhbb\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.509051 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.508990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-dshm\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.509051 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.509013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tls-certs\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.509157 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.509053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-home\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.509157 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.509073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tmp-dir\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.509465 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.509443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-model-cache\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.509517 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.509482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-home\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.509613 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.509511    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tmp-dir\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.511370 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.511339    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-dshm\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.511591 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.511570    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tls-certs\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.526122 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.526094    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swhbb\" (UniqueName: \"kubernetes.io/projected/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-kube-api-access-swhbb\") pod \"e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") " pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.624020 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.623975    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:04.906875 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:04.906841    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"]
Sep 06 10:33:04.908286 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:33:04.908257    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e0a10ec_8a7d_41c6_b3f6_ccb6536aa1f9.slice/crio-9b59e960def2f56ac2f38e1130539f7c9a31e8502a7cf8c6b3f3981259175428 WatchSource:0}: Error finding container 9b59e960def2f56ac2f38e1130539f7c9a31e8502a7cf8c6b3f3981259175428: Status 404 returned error can't find the container with id 9b59e960def2f56ac2f38e1130539f7c9a31e8502a7cf8c6b3f3981259175428
Sep 06 10:33:05.247959 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:05.247842    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg" event={"ID":"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9","Type":"ContainerStarted","Data":"b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481"}
Sep 06 10:33:05.248136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:05.248015    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:05.248136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:05.248066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg" event={"ID":"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9","Type":"ContainerStarted","Data":"9b59e960def2f56ac2f38e1130539f7c9a31e8502a7cf8c6b3f3981259175428"}
Sep 06 10:33:05.249242 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:05.249198    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:05.264467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:05.264411    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg" podStartSLOduration=1.264396394 podStartE2EDuration="1.264396394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:05.263621777 +0000 UTC m=+673.747121922" watchObservedRunningTime="2026-09-06 10:33:05.264396394 +0000 UTC m=+673.747896503"
Sep 06 10:33:06.254844 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:06.254804    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6" event={"ID":"b86e1700-1ee3-460e-969c-9593f94bead3","Type":"ContainerStarted","Data":"7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad"}
Sep 06 10:33:06.275034 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:06.274979    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6" podStartSLOduration=2.881192682 podStartE2EDuration="4.274954112s" totalImagesPullingTime="1.39376143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:06.272945085 +0000 UTC m=+674.756445211" watchObservedRunningTime="2026-09-06 10:33:06.274954112 +0000 UTC m=+674.758454221"
Sep 06 10:33:07.262279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:07.262242    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:08.092925 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.092888    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"]
Sep 06 10:33:08.097042 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.097018    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:33:08.099903 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.099877    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-5dda3d\""
Sep 06 10:33:08.100039 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.099930    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-5dda3d-dockercfg-v5cbx\""
Sep 06 10:33:08.110499 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.110476    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"]
Sep 06 10:33:08.249240 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.249158    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9324d6f4-6547-435b-b966-94fe1e6fc6ad-plugins-config-volume\") pod \"payload-processing-e2e-autores-5dda3d-845577577f-42fdp\" (UID: \"9324d6f4-6547-435b-b966-94fe1e6fc6ad\") " pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:33:08.249446 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.249263    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxjhm\" (UniqueName: \"kubernetes.io/projected/9324d6f4-6547-435b-b966-94fe1e6fc6ad-kube-api-access-jxjhm\") pod \"payload-processing-e2e-autores-5dda3d-845577577f-42fdp\" (UID: \"9324d6f4-6547-435b-b966-94fe1e6fc6ad\") " pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:33:08.350626 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.350525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9324d6f4-6547-435b-b966-94fe1e6fc6ad-plugins-config-volume\") pod \"payload-processing-e2e-autores-5dda3d-845577577f-42fdp\" (UID: \"9324d6f4-6547-435b-b966-94fe1e6fc6ad\") " pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:33:08.351089 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.350673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxjhm\" (UniqueName: \"kubernetes.io/projected/9324d6f4-6547-435b-b966-94fe1e6fc6ad-kube-api-access-jxjhm\") pod \"payload-processing-e2e-autores-5dda3d-845577577f-42fdp\" (UID: \"9324d6f4-6547-435b-b966-94fe1e6fc6ad\") " pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:33:08.351341 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.351323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9324d6f4-6547-435b-b966-94fe1e6fc6ad-plugins-config-volume\") pod \"payload-processing-e2e-autores-5dda3d-845577577f-42fdp\" (UID: \"9324d6f4-6547-435b-b966-94fe1e6fc6ad\") " pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:33:08.358906 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.358875    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxjhm\" (UniqueName: \"kubernetes.io/projected/9324d6f4-6547-435b-b966-94fe1e6fc6ad-kube-api-access-jxjhm\") pod \"payload-processing-e2e-autores-5dda3d-845577577f-42fdp\" (UID: \"9324d6f4-6547-435b-b966-94fe1e6fc6ad\") " pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:33:08.408516 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.408477    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:33:08.697111 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.697082    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"]
Sep 06 10:33:08.700193 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:33:08.700157    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9324d6f4_6547_435b_b966_94fe1e6fc6ad.slice/crio-943c9fd14a24e753b024aa0e614a56868eb631cfe20f0250ff04ab05988d9e3d WatchSource:0}: Error finding container 943c9fd14a24e753b024aa0e614a56868eb631cfe20f0250ff04ab05988d9e3d: Status 404 returned error can't find the container with id 943c9fd14a24e753b024aa0e614a56868eb631cfe20f0250ff04ab05988d9e3d
Sep 06 10:33:08.702643 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:08.702628    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:33:09.274090 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:09.274051    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp" event={"ID":"9324d6f4-6547-435b-b966-94fe1e6fc6ad","Type":"ContainerStarted","Data":"943c9fd14a24e753b024aa0e614a56868eb631cfe20f0250ff04ab05988d9e3d"}
Sep 06 10:33:11.286279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:11.286246    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp" event={"ID":"9324d6f4-6547-435b-b966-94fe1e6fc6ad","Type":"ContainerStarted","Data":"134aa8103ce9edbd4ffc44bd3cbe51085998982a1cd524fad514c1df96e78e6e"}
Sep 06 10:33:11.286666 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:11.286363    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:33:11.305179 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:11.305122    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp" podStartSLOduration=1.012630883 podStartE2EDuration="3.305105064s" totalImagesPullingTime="2.292474181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:11.303286724 +0000 UTC m=+679.786786854" watchObservedRunningTime="2026-09-06 10:33:11.305105064 +0000 UTC m=+679.788605173"
Sep 06 10:33:13.272031 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:13.271995    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:14.624506 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:14.624460    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:14.626166 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:14.626144    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:14.639611 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:14.639577    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:15.339668 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:15.339634    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:33:21.289404 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:21.289360    2564 patch_prober.go:28] interesting pod/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 10:33:21.289825 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:21.289432    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 10:33:22.293588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:22.293554    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:33:23.368675 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:23.368590    2564 generic.go:356] "Generic (PLEG): container finished" podID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerID="6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318" exitCode=0
Sep 06 10:33:23.369063 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:23.368665    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" event={"ID":"4104a875-ff2b-432e-af78-1ed71b7fdd7b","Type":"ContainerDied","Data":"6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318"}
Sep 06 10:33:23.369141 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:23.369126    2564 scope.go:117] "RemoveContainer" containerID="6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318"
Sep 06 10:33:23.371150 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:23.371113    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:33:23.371282 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:23.371205    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:33:23.371282 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:23.371269    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:33:24.100274 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.100237    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"]
Sep 06 10:33:24.107534 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.107497    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.113917 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.113868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"]
Sep 06 10:33:24.115503 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.114419    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bd3407-serving-cert\""
Sep 06 10:33:24.222747 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.222697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-maas-api-tls\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.223140 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.223115    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-metrics-certs\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.223329 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.223304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4642f16-18cb-41c9-a3fc-569a96bf7db0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.223485 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.223464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb88q\" (UniqueName: \"kubernetes.io/projected/e4642f16-18cb-41c9-a3fc-569a96bf7db0-kube-api-access-mb88q\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.324176 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.324131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-metrics-certs\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.328389 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.328344    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4642f16-18cb-41c9-a3fc-569a96bf7db0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.328639 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.328612    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-metrics-certs\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.329068 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.329045    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb88q\" (UniqueName: \"kubernetes.io/projected/e4642f16-18cb-41c9-a3fc-569a96bf7db0-kube-api-access-mb88q\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.329297 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.329274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-maas-api-tls\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.329390 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.329357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4642f16-18cb-41c9-a3fc-569a96bf7db0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.332015 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.331986    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-maas-api-tls\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.336960 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.336937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb88q\" (UniqueName: \"kubernetes.io/projected/e4642f16-18cb-41c9-a3fc-569a96bf7db0-kube-api-access-mb88q\") pod \"maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.376751 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.376673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" event={"ID":"4104a875-ff2b-432e-af78-1ed71b7fdd7b","Type":"ContainerStarted","Data":"9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c"}
Sep 06 10:33:24.442161 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.442125    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:24.485765 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.485719    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:24.485997 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.485973    2564 patch_prober.go:28] interesting pod/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 06 10:33:24.486092 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.486027    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 06 10:33:24.721320 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:24.720299    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"]
Sep 06 10:33:24.733543 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:33:24.733500    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4642f16_18cb_41c9_a3fc_569a96bf7db0.slice/crio-5dbdff79447ce51947fe1c2c9650b57aa258b75805d21ee7fd0ca0ffbb371577 WatchSource:0}: Error finding container 5dbdff79447ce51947fe1c2c9650b57aa258b75805d21ee7fd0ca0ffbb371577: Status 404 returned error can't find the container with id 5dbdff79447ce51947fe1c2c9650b57aa258b75805d21ee7fd0ca0ffbb371577
Sep 06 10:33:25.383053 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:25.382945    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw" event={"ID":"e4642f16-18cb-41c9-a3fc-569a96bf7db0","Type":"ContainerStarted","Data":"77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050"}
Sep 06 10:33:25.383485 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:25.383248    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:25.383485 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:25.383298    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw" event={"ID":"e4642f16-18cb-41c9-a3fc-569a96bf7db0","Type":"ContainerStarted","Data":"5dbdff79447ce51947fe1c2c9650b57aa258b75805d21ee7fd0ca0ffbb371577"}
Sep 06 10:33:25.399614 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:25.399566    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw" podStartSLOduration=1.069986379 podStartE2EDuration="1.399550248s" totalImagesPullingTime="329.563869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:25.398628687 +0000 UTC m=+693.882128817" watchObservedRunningTime="2026-09-06 10:33:25.399550248 +0000 UTC m=+693.883050356"
Sep 06 10:33:25.485858 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:25.485826    2564 patch_prober.go:28] interesting pod/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 06 10:33:25.486013 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:25.485879    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 06 10:33:26.486224 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:26.486178    2564 patch_prober.go:28] interesting pod/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 06 10:33:26.486604 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:26.486261    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 06 10:33:27.486315 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:27.486277    2564 patch_prober.go:28] interesting pod/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 06 10:33:27.486691 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:27.486345    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 06 10:33:28.486565 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:28.486526    2564 patch_prober.go:28] interesting pod/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 06 10:33:28.486975 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:28.486598    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 06 10:33:29.656391 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:29.656360    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:29.657005 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:29.656642    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:29.657752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:29.657732    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:33:31.394930 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:31.394898    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:33:43.108771 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.108734    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"]
Sep 06 10:33:43.109312 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.108958    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6" podUID="b86e1700-1ee3-460e-969c-9593f94bead3" containerName="maas-api" containerID="cri-o://7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad" gracePeriod=30
Sep 06 10:33:43.267257 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.267185    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6" podUID="b86e1700-1ee3-460e-969c-9593f94bead3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.36:8443/health\": dial tcp 10.134.0.36:8443: connect: connection refused"
Sep 06 10:33:43.360146 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.360076    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:43.416532 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.416490    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-metrics-certs\") pod \"b86e1700-1ee3-460e-969c-9593f94bead3\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") "
Sep 06 10:33:43.416711 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.416553    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls\") pod \"b86e1700-1ee3-460e-969c-9593f94bead3\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") "
Sep 06 10:33:43.416711 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.416609    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86e1700-1ee3-460e-969c-9593f94bead3-openshift-service-ca\") pod \"b86e1700-1ee3-460e-969c-9593f94bead3\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") "
Sep 06 10:33:43.416711 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.416693    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd5h8\" (UniqueName: \"kubernetes.io/projected/b86e1700-1ee3-460e-969c-9593f94bead3-kube-api-access-bd5h8\") pod \"b86e1700-1ee3-460e-969c-9593f94bead3\" (UID: \"b86e1700-1ee3-460e-969c-9593f94bead3\") "
Sep 06 10:33:43.417049 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.417018    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86e1700-1ee3-460e-969c-9593f94bead3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b86e1700-1ee3-460e-969c-9593f94bead3" (UID: "b86e1700-1ee3-460e-969c-9593f94bead3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:33:43.418894 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.418863    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b86e1700-1ee3-460e-969c-9593f94bead3" (UID: "b86e1700-1ee3-460e-969c-9593f94bead3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:33:43.419004 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.418904    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b86e1700-1ee3-460e-969c-9593f94bead3" (UID: "b86e1700-1ee3-460e-969c-9593f94bead3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:33:43.419004 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.418926    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86e1700-1ee3-460e-969c-9593f94bead3-kube-api-access-bd5h8" (OuterVolumeSpecName: "kube-api-access-bd5h8") pod "b86e1700-1ee3-460e-969c-9593f94bead3" (UID: "b86e1700-1ee3-460e-969c-9593f94bead3"). InnerVolumeSpecName "kube-api-access-bd5h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:33:43.479135 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.479095    2564 generic.go:356] "Generic (PLEG): container finished" podID="b86e1700-1ee3-460e-969c-9593f94bead3" containerID="7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad" exitCode=0
Sep 06 10:33:43.479337 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.479171    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"
Sep 06 10:33:43.479337 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.479188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6" event={"ID":"b86e1700-1ee3-460e-969c-9593f94bead3","Type":"ContainerDied","Data":"7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad"}
Sep 06 10:33:43.479337 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.479323    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6" event={"ID":"b86e1700-1ee3-460e-969c-9593f94bead3","Type":"ContainerDied","Data":"7bc90a4922e6e32a636f54e2139dca4d8857f9afe1c7387cf9654ecb16c5bc2e"}
Sep 06 10:33:43.479544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.479356    2564 scope.go:117] "RemoveContainer" containerID="7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad"
Sep 06 10:33:43.493294 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.493254    2564 scope.go:117] "RemoveContainer" containerID="7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad"
Sep 06 10:33:43.494174 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:33:43.494147    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad\": container with ID starting with 7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad not found: ID does not exist" containerID="7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad"
Sep 06 10:33:43.494379 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.494186    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad"} err="failed to get container status \"7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad\": rpc error: code = NotFound desc = could not find container \"7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad\": container with ID starting with 7ba0719c099c11e499f42b6b048050bd7164ce0e5715d3caa1b0bc6c4f91baad not found: ID does not exist"
Sep 06 10:33:43.506730 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.506428    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"]
Sep 06 10:33:43.509410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.509382    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbaf75b1-878cb58b4-rdbk6"]
Sep 06 10:33:43.517812 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.517790    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd5h8\" (UniqueName: \"kubernetes.io/projected/b86e1700-1ee3-460e-969c-9593f94bead3-kube-api-access-bd5h8\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:33:43.517812 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.517814    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-metrics-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:33:43.517961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.517830    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e1700-1ee3-460e-969c-9593f94bead3-maas-api-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:33:43.517961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:43.517842    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86e1700-1ee3-460e-969c-9593f94bead3-openshift-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:33:44.032251 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.030973    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b86e1700-1ee3-460e-969c-9593f94bead3" path="/var/lib/kubelet/pods/b86e1700-1ee3-460e-969c-9593f94bead3/volumes"
Sep 06 10:33:44.514340 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.514304    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"]
Sep 06 10:33:44.514919 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.514898    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86e1700-1ee3-460e-969c-9593f94bead3" containerName="maas-api"
Sep 06 10:33:44.514970 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.514924    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86e1700-1ee3-460e-969c-9593f94bead3" containerName="maas-api"
Sep 06 10:33:44.515070 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.515058    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86e1700-1ee3-460e-969c-9593f94bead3" containerName="maas-api"
Sep 06 10:33:44.519887 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.519864    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.522498 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.522478    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ca3c8694-gw-openshift-default-dockercfg-btp8v\""
Sep 06 10:33:44.535902 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.535880    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"]
Sep 06 10:33:44.627955 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.627918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-credential-socket\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.627955 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.627962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-socket\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.628237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.627998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-data\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.628237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.628033    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6vhs\" (UniqueName: \"kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-kube-api-access-g6vhs\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.628237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.628109    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istiod-ca-cert\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.628237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.628148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-token\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.628448 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.628265    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-podinfo\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.628448 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.628288    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-certs\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.628448 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.628303    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-envoy\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.729338 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.729292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istiod-ca-cert\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.729338 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.729340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-token\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.729605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.729377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-podinfo\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.729605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.729395    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-certs\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.729605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.729411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-envoy\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.729605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.729518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-credential-socket\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.729605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.729568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-socket\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.729846 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.729620    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-data\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.729846 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.729659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6vhs\" (UniqueName: \"kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-kube-api-access-g6vhs\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.729944 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.729896    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-certs\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.730098 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.730075    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-credential-socket\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.730153 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.730126    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istiod-ca-cert\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.730312 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.730289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-socket\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.730378 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.730323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-data\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.731940 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.731923    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-envoy\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.732041 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.732025    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-podinfo\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.737374 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.737347    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-token\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.737533 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.737516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6vhs\" (UniqueName: \"kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-kube-api-access-g6vhs\") pod \"e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") " pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:44.829951 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:44.829915    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:45.108642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:45.108617    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"]
Sep 06 10:33:45.110462 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:33:45.110437    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6708a5_0ae5_4dc9_a9ab_05e4308b4c16.slice/crio-be3bb7359b1f281dae6b4fe44709ce20f4b8670990576c349b8fc8eeacd01a5d WatchSource:0}: Error finding container be3bb7359b1f281dae6b4fe44709ce20f4b8670990576c349b8fc8eeacd01a5d: Status 404 returned error can't find the container with id be3bb7359b1f281dae6b4fe44709ce20f4b8670990576c349b8fc8eeacd01a5d
Sep 06 10:33:45.113136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:45.113097    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:33:45.113230 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:45.113193    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:33:45.113284 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:45.113264    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:33:45.492949 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:45.492851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" event={"ID":"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16","Type":"ContainerStarted","Data":"80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7"}
Sep 06 10:33:45.492949 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:45.492915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" event={"ID":"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16","Type":"ContainerStarted","Data":"be3bb7359b1f281dae6b4fe44709ce20f4b8670990576c349b8fc8eeacd01a5d"}
Sep 06 10:33:45.516055 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:45.515999    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" podStartSLOduration=1.515981587 podStartE2EDuration="1.515981587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:45.514955662 +0000 UTC m=+713.998455794" watchObservedRunningTime="2026-09-06 10:33:45.515981587 +0000 UTC m=+713.999481707"
Sep 06 10:33:45.830466 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:45.830425    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:46.835615 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:46.835567    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:47.507609 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:47.507579    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:47.508586 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:47.508568    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:33:48.021946 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.021909    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"]
Sep 06 10:33:48.025503 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.025482    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:33:48.028428 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.028402    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ca3c8694\""
Sep 06 10:33:48.041913 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.041890    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"]
Sep 06 10:33:48.075872 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.075839    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"]
Sep 06 10:33:48.079468 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.079432    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:33:48.082487 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.082466    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ca3c8694-dockercfg-qpz9j\""
Sep 06 10:33:48.100436 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.100406    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"]
Sep 06 10:33:48.161348 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.161311    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw\" (UID: \"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:33:48.161530 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.161351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztnvb\" (UniqueName: \"kubernetes.io/projected/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-kube-api-access-ztnvb\") pod \"payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw\" (UID: \"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:33:48.161607 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.161523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efceea9-8536-4161-968d-82e81c55c20e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb\" (UID: \"8efceea9-8536-4161-968d-82e81c55c20e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:33:48.161607 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.161590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgrb\" (UniqueName: \"kubernetes.io/projected/8efceea9-8536-4161-968d-82e81c55c20e-kube-api-access-zbgrb\") pod \"payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb\" (UID: \"8efceea9-8536-4161-968d-82e81c55c20e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:33:48.262659 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.262618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efceea9-8536-4161-968d-82e81c55c20e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb\" (UID: \"8efceea9-8536-4161-968d-82e81c55c20e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:33:48.262862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.262688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgrb\" (UniqueName: \"kubernetes.io/projected/8efceea9-8536-4161-968d-82e81c55c20e-kube-api-access-zbgrb\") pod \"payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb\" (UID: \"8efceea9-8536-4161-968d-82e81c55c20e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:33:48.262862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.262750    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw\" (UID: \"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:33:48.262862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.262767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztnvb\" (UniqueName: \"kubernetes.io/projected/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-kube-api-access-ztnvb\") pod \"payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw\" (UID: \"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:33:48.263470 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.263445    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw\" (UID: \"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:33:48.263534 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.263448    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efceea9-8536-4161-968d-82e81c55c20e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb\" (UID: \"8efceea9-8536-4161-968d-82e81c55c20e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:33:48.271433 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.271402    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztnvb\" (UniqueName: \"kubernetes.io/projected/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-kube-api-access-ztnvb\") pod \"payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw\" (UID: \"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4\") " pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:33:48.271554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.271406    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgrb\" (UniqueName: \"kubernetes.io/projected/8efceea9-8536-4161-968d-82e81c55c20e-kube-api-access-zbgrb\") pod \"payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb\" (UID: \"8efceea9-8536-4161-968d-82e81c55c20e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:33:48.337605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.337565    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:33:48.390197 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.390164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:33:48.612353 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.612328    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"]
Sep 06 10:33:48.614108 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:33:48.614079    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efceea9_8536_4161_968d_82e81c55c20e.slice/crio-a7b588a5d6a1a69626ba03c330c0e2e27a70394bf9b60eca4451d6944054e8df WatchSource:0}: Error finding container a7b588a5d6a1a69626ba03c330c0e2e27a70394bf9b60eca4451d6944054e8df: Status 404 returned error can't find the container with id a7b588a5d6a1a69626ba03c330c0e2e27a70394bf9b60eca4451d6944054e8df
Sep 06 10:33:48.676981 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:48.676890    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"]
Sep 06 10:33:48.679754 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:33:48.679729    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3bc871_8c6f_4ab0_80fc_f61cf204ace4.slice/crio-c15d456ab6ad1a43f10afd53ad07ab38b112a7ca426de0bb1ecb59f3c06bc3ed WatchSource:0}: Error finding container c15d456ab6ad1a43f10afd53ad07ab38b112a7ca426de0bb1ecb59f3c06bc3ed: Status 404 returned error can't find the container with id c15d456ab6ad1a43f10afd53ad07ab38b112a7ca426de0bb1ecb59f3c06bc3ed
Sep 06 10:33:49.519440 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:49.519403    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw" event={"ID":"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4","Type":"ContainerStarted","Data":"39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39"}
Sep 06 10:33:49.519862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:49.519473    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:33:49.519862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:49.519496    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw" event={"ID":"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4","Type":"ContainerStarted","Data":"c15d456ab6ad1a43f10afd53ad07ab38b112a7ca426de0bb1ecb59f3c06bc3ed"}
Sep 06 10:33:49.521089 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:49.521052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb" event={"ID":"8efceea9-8536-4161-968d-82e81c55c20e","Type":"ContainerStarted","Data":"a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316"}
Sep 06 10:33:49.521191 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:49.521110    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb" event={"ID":"8efceea9-8536-4161-968d-82e81c55c20e","Type":"ContainerStarted","Data":"a7b588a5d6a1a69626ba03c330c0e2e27a70394bf9b60eca4451d6944054e8df"}
Sep 06 10:33:49.536433 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:49.536382    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw" podStartSLOduration=1.536363396 podStartE2EDuration="1.536363396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:49.535873136 +0000 UTC m=+718.019373275" watchObservedRunningTime="2026-09-06 10:33:49.536363396 +0000 UTC m=+718.019863509"
Sep 06 10:33:49.553089 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:49.553038    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb" podStartSLOduration=2.553022327 podStartE2EDuration="2.553022327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:49.551587912 +0000 UTC m=+718.035088056" watchObservedRunningTime="2026-09-06 10:33:49.553022327 +0000 UTC m=+718.036522436"
Sep 06 10:33:50.527334 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:33:50.527301    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:34:00.529844 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:00.529811    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:34:01.534791 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:01.534756    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:34:07.616377 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:07.616342    2564 generic.go:356] "Generic (PLEG): container finished" podID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerID="80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7" exitCode=0
Sep 06 10:34:07.616854 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:07.616424    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" event={"ID":"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16","Type":"ContainerDied","Data":"80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7"}
Sep 06 10:34:07.616854 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:07.616845    2564 scope.go:117] "RemoveContainer" containerID="80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7"
Sep 06 10:34:07.617984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:07.617952    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:34:07.618089 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:07.618017    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:34:07.618089 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:07.618051    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:34:08.629644 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:08.629607    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" event={"ID":"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16","Type":"ContainerStarted","Data":"491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7"}
Sep 06 10:34:08.830960 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:08.830911    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:34:08.831249 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:08.831204    2564 patch_prober.go:28] interesting pod/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 06 10:34:08.831360 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:08.831289    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 06 10:34:09.831091 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:09.831049    2564 patch_prober.go:28] interesting pod/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 06 10:34:09.831531 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:09.831113    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 06 10:34:10.831309 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:10.831264    2564 patch_prober.go:28] interesting pod/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 06 10:34:10.831721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:10.831332    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 06 10:34:11.830686 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:11.830641    2564 patch_prober.go:28] interesting pod/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 06 10:34:11.830887 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:11.830718    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 06 10:34:12.830865 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:12.830822    2564 patch_prober.go:28] interesting pod/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 06 10:34:12.831335 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:12.830899    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 06 10:34:13.831028 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:13.830991    2564 patch_prober.go:28] interesting pod/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 06 10:34:13.831456 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:13.831051    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 06 10:34:14.835076 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:14.835044    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:34:14.835686 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:14.835335    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:34:14.835952 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:14.835932    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:34:18.844002 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:18.843967    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"]
Sep 06 10:34:18.844526 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:18.844203    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw" podUID="5a3bc871-8c6f-4ab0-80fc-f61cf204ace4" containerName="payload-processing" containerID="cri-o://39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39" gracePeriod=30
Sep 06 10:34:18.851844 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:18.851810    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"]
Sep 06 10:34:18.852109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:18.852039    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb" podUID="8efceea9-8536-4161-968d-82e81c55c20e" containerName="payload-pre-processing" containerID="cri-o://a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316" gracePeriod=30
Sep 06 10:34:19.120603 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.120574    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:34:19.148760 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.148737    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:34:19.187916 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.187880    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"]
Sep 06 10:34:19.188141 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.188112    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy" containerID="cri-o://491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7" gracePeriod=30
Sep 06 10:34:19.192117 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.191397    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztnvb\" (UniqueName: \"kubernetes.io/projected/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-kube-api-access-ztnvb\") pod \"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4\" (UID: \"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4\") "
Sep 06 10:34:19.192493 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.192447    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a3bc871-8c6f-4ab0-80fc-f61cf204ace4" (UID: "5a3bc871-8c6f-4ab0-80fc-f61cf204ace4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:34:19.192615 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.191963    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-plugins-config-volume\") pod \"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4\" (UID: \"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4\") "
Sep 06 10:34:19.192980 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.192945    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:19.196474 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.196429    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-kube-api-access-ztnvb" (OuterVolumeSpecName: "kube-api-access-ztnvb") pod "5a3bc871-8c6f-4ab0-80fc-f61cf204ace4" (UID: "5a3bc871-8c6f-4ab0-80fc-f61cf204ace4"). InnerVolumeSpecName "kube-api-access-ztnvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:34:19.294140 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.294103    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efceea9-8536-4161-968d-82e81c55c20e-plugins-config-volume\") pod \"8efceea9-8536-4161-968d-82e81c55c20e\" (UID: \"8efceea9-8536-4161-968d-82e81c55c20e\") "
Sep 06 10:34:19.294366 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.294176    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbgrb\" (UniqueName: \"kubernetes.io/projected/8efceea9-8536-4161-968d-82e81c55c20e-kube-api-access-zbgrb\") pod \"8efceea9-8536-4161-968d-82e81c55c20e\" (UID: \"8efceea9-8536-4161-968d-82e81c55c20e\") "
Sep 06 10:34:19.294594 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.294567    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efceea9-8536-4161-968d-82e81c55c20e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8efceea9-8536-4161-968d-82e81c55c20e" (UID: "8efceea9-8536-4161-968d-82e81c55c20e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:34:19.294594 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.294583    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztnvb\" (UniqueName: \"kubernetes.io/projected/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4-kube-api-access-ztnvb\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:19.296467 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.296428    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efceea9-8536-4161-968d-82e81c55c20e-kube-api-access-zbgrb" (OuterVolumeSpecName: "kube-api-access-zbgrb") pod "8efceea9-8536-4161-968d-82e81c55c20e" (UID: "8efceea9-8536-4161-968d-82e81c55c20e"). InnerVolumeSpecName "kube-api-access-zbgrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:34:19.395731 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.395634    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efceea9-8536-4161-968d-82e81c55c20e-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:19.395731 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.395669    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbgrb\" (UniqueName: \"kubernetes.io/projected/8efceea9-8536-4161-968d-82e81c55c20e-kube-api-access-zbgrb\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:19.686378 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.686289    2564 generic.go:356] "Generic (PLEG): container finished" podID="5a3bc871-8c6f-4ab0-80fc-f61cf204ace4" containerID="39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39" exitCode=0
Sep 06 10:34:19.686378 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.686333    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"
Sep 06 10:34:19.686588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.686371    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw" event={"ID":"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4","Type":"ContainerDied","Data":"39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39"}
Sep 06 10:34:19.686588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.686421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw" event={"ID":"5a3bc871-8c6f-4ab0-80fc-f61cf204ace4","Type":"ContainerDied","Data":"c15d456ab6ad1a43f10afd53ad07ab38b112a7ca426de0bb1ecb59f3c06bc3ed"}
Sep 06 10:34:19.686588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.686453    2564 scope.go:117] "RemoveContainer" containerID="39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39"
Sep 06 10:34:19.687862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.687841    2564 generic.go:356] "Generic (PLEG): container finished" podID="8efceea9-8536-4161-968d-82e81c55c20e" containerID="a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316" exitCode=0
Sep 06 10:34:19.687979 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.687888    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"
Sep 06 10:34:19.687979 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.687911    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb" event={"ID":"8efceea9-8536-4161-968d-82e81c55c20e","Type":"ContainerDied","Data":"a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316"}
Sep 06 10:34:19.687979 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.687951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb" event={"ID":"8efceea9-8536-4161-968d-82e81c55c20e","Type":"ContainerDied","Data":"a7b588a5d6a1a69626ba03c330c0e2e27a70394bf9b60eca4451d6944054e8df"}
Sep 06 10:34:19.695878 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.695840    2564 scope.go:117] "RemoveContainer" containerID="39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39"
Sep 06 10:34:19.696272 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:34:19.696242    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39\": container with ID starting with 39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39 not found: ID does not exist" containerID="39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39"
Sep 06 10:34:19.696373 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.696279    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39"} err="failed to get container status \"39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39\": rpc error: code = NotFound desc = could not find container \"39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39\": container with ID starting with 39bd81cf303d67253050d176fc9f3ac114372c98fb4f9607b22e0b017b88ef39 not found: ID does not exist"
Sep 06 10:34:19.696373 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.696301    2564 scope.go:117] "RemoveContainer" containerID="a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316"
Sep 06 10:34:19.707580 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.705417    2564 scope.go:117] "RemoveContainer" containerID="a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316"
Sep 06 10:34:19.707726 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:34:19.707683    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316\": container with ID starting with a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316 not found: ID does not exist" containerID="a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316"
Sep 06 10:34:19.707784 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.707715    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316"} err="failed to get container status \"a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316\": rpc error: code = NotFound desc = could not find container \"a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316\": container with ID starting with a7d6ffe221f7beda7bd10059faba3ac80ec4daee825016ffc0d614fc97812316 not found: ID does not exist"
Sep 06 10:34:19.713524 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.713500    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"]
Sep 06 10:34:19.717596 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.717574    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ca3c8694-6d567cb4fd-vc6jb"]
Sep 06 10:34:19.730822 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.730797    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"]
Sep 06 10:34:19.734442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:19.734421    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ca3c8694-f4d9dddf4-pzvkw"]
Sep 06 10:34:20.029438 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:20.029352    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a3bc871-8c6f-4ab0-80fc-f61cf204ace4" path="/var/lib/kubelet/pods/5a3bc871-8c6f-4ab0-80fc-f61cf204ace4/volumes"
Sep 06 10:34:20.029793 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:20.029715    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8efceea9-8536-4161-968d-82e81c55c20e" path="/var/lib/kubelet/pods/8efceea9-8536-4161-968d-82e81c55c20e/volumes"
Sep 06 10:34:23.685442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.685405    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"]
Sep 06 10:34:23.686061 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.686045    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a3bc871-8c6f-4ab0-80fc-f61cf204ace4" containerName="payload-processing"
Sep 06 10:34:23.686109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.686063    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a3bc871-8c6f-4ab0-80fc-f61cf204ace4" containerName="payload-processing"
Sep 06 10:34:23.686109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.686081    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8efceea9-8536-4161-968d-82e81c55c20e" containerName="payload-pre-processing"
Sep 06 10:34:23.686109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.686090    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8efceea9-8536-4161-968d-82e81c55c20e" containerName="payload-pre-processing"
Sep 06 10:34:23.686232 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.686205    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a3bc871-8c6f-4ab0-80fc-f61cf204ace4" containerName="payload-processing"
Sep 06 10:34:23.686278 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.686236    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8efceea9-8536-4161-968d-82e81c55c20e" containerName="payload-pre-processing"
Sep 06 10:34:23.689761 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.689738    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.692509 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.692488    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-17db6c23-serving-cert\""
Sep 06 10:34:23.701112 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.701088    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"]
Sep 06 10:34:23.842895 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.842860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw5nm\" (UniqueName: \"kubernetes.io/projected/8080509e-d8a3-4d19-b91b-8c117b5e1e90-kube-api-access-rw5nm\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.843078 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.842912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8080509e-d8a3-4d19-b91b-8c117b5e1e90-openshift-service-ca\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.843078 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.843015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-maas-api-tls\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.843078 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.843048    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-metrics-certs\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.944160 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.944063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8080509e-d8a3-4d19-b91b-8c117b5e1e90-openshift-service-ca\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.944160 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.944158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-maas-api-tls\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.944373 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.944187    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-metrics-certs\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.944373 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.944312    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5nm\" (UniqueName: \"kubernetes.io/projected/8080509e-d8a3-4d19-b91b-8c117b5e1e90-kube-api-access-rw5nm\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.944788 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.944767    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8080509e-d8a3-4d19-b91b-8c117b5e1e90-openshift-service-ca\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.946656 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.946634    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-metrics-certs\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.946821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.946799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-maas-api-tls\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:23.952355 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:23.952335    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5nm\" (UniqueName: \"kubernetes.io/projected/8080509e-d8a3-4d19-b91b-8c117b5e1e90-kube-api-access-rw5nm\") pod \"maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:24.004434 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.004396    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:24.277101 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.277073    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"]
Sep 06 10:34:24.279031 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:34:24.279002    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8080509e_d8a3_4d19_b91b_8c117b5e1e90.slice/crio-ff64d3a4f2c55022e207c8953f1faab1e4067b7dcb6e0405d826b1390f1dde66 WatchSource:0}: Error finding container ff64d3a4f2c55022e207c8953f1faab1e4067b7dcb6e0405d826b1390f1dde66: Status 404 returned error can't find the container with id ff64d3a4f2c55022e207c8953f1faab1e4067b7dcb6e0405d826b1390f1dde66
Sep 06 10:34:24.354077 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.354000    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"]
Sep 06 10:34:24.356637 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.356617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:24.359266 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.359238    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-17db6c23-dockercfg-ndrnq\""
Sep 06 10:34:24.359266 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.359259    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-17db6c23\""
Sep 06 10:34:24.378737 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.378048    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"]
Sep 06 10:34:24.436624 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.436596    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:34:24.449050 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.449019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c1a055-6a41-497b-bd39-18bfd4001797-plugins-config-volume\") pod \"payload-processing-e2e-ait-17db6c23-c9d564899-vph8p\" (UID: \"56c1a055-6a41-497b-bd39-18bfd4001797\") " pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:24.449260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.449088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-schrq\" (UniqueName: \"kubernetes.io/projected/56c1a055-6a41-497b-bd39-18bfd4001797-kube-api-access-schrq\") pod \"payload-processing-e2e-ait-17db6c23-c9d564899-vph8p\" (UID: \"56c1a055-6a41-497b-bd39-18bfd4001797\") " pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:24.549532 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549497    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6vhs\" (UniqueName: \"kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-kube-api-access-g6vhs\") pod \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") "
Sep 06 10:34:24.549721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549562    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-data\") pod \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") "
Sep 06 10:34:24.549721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549583    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-token\") pod \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") "
Sep 06 10:34:24.549721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549600    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-envoy\") pod \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") "
Sep 06 10:34:24.549721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549627    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istiod-ca-cert\") pod \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") "
Sep 06 10:34:24.549721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-credential-socket\") pod \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") "
Sep 06 10:34:24.549721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549671    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-certs\") pod \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") "
Sep 06 10:34:24.550042 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549738    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-podinfo\") pod \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") "
Sep 06 10:34:24.550042 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549793    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-socket\") pod \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\" (UID: \"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16\") "
Sep 06 10:34:24.550042 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549955    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-schrq\" (UniqueName: \"kubernetes.io/projected/56c1a055-6a41-497b-bd39-18bfd4001797-kube-api-access-schrq\") pod \"payload-processing-e2e-ait-17db6c23-c9d564899-vph8p\" (UID: \"56c1a055-6a41-497b-bd39-18bfd4001797\") " pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:24.550042 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549972    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" (UID: "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:34:24.550266 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.550068    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" (UID: "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:34:24.550266 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.549985    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-data" (OuterVolumeSpecName: "istio-data") pod "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" (UID: "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:34:24.550406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.550174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c1a055-6a41-497b-bd39-18bfd4001797-plugins-config-volume\") pod \"payload-processing-e2e-ait-17db6c23-c9d564899-vph8p\" (UID: \"56c1a055-6a41-497b-bd39-18bfd4001797\") " pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:24.550406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.550285    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" (UID: "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:34:24.550406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.550321    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" (UID: "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:34:24.550555 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.550437    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-socket\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:24.550555 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.550457    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-data\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:24.550555 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.550473    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istiod-ca-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:24.550555 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.550486    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-credential-socket\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:24.550555 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.550500    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-workload-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:24.551152 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.551124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c1a055-6a41-497b-bd39-18bfd4001797-plugins-config-volume\") pod \"payload-processing-e2e-ait-17db6c23-c9d564899-vph8p\" (UID: \"56c1a055-6a41-497b-bd39-18bfd4001797\") " pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:24.551936 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.551910    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" (UID: "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 10:34:24.552012 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.551967    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-kube-api-access-g6vhs" (OuterVolumeSpecName: "kube-api-access-g6vhs") pod "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" (UID: "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16"). InnerVolumeSpecName "kube-api-access-g6vhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:34:24.552012 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.551992    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-token" (OuterVolumeSpecName: "istio-token") pod "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" (UID: "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:34:24.552190 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.552176    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" (UID: "af6708a5-0ae5-4dc9-a9ab-05e4308b4c16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:34:24.558176 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.558158    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-schrq\" (UniqueName: \"kubernetes.io/projected/56c1a055-6a41-497b-bd39-18bfd4001797-kube-api-access-schrq\") pod \"payload-processing-e2e-ait-17db6c23-c9d564899-vph8p\" (UID: \"56c1a055-6a41-497b-bd39-18bfd4001797\") " pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:24.651610 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.651575    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6vhs\" (UniqueName: \"kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-kube-api-access-g6vhs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:24.651610 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.651607    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-token\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:24.651610 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.651616    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-envoy\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:24.651844 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.651623    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16-istio-podinfo\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:24.680041 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.680006    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:24.718089 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.718061    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4" event={"ID":"8080509e-d8a3-4d19-b91b-8c117b5e1e90","Type":"ContainerStarted","Data":"ff64d3a4f2c55022e207c8953f1faab1e4067b7dcb6e0405d826b1390f1dde66"}
Sep 06 10:34:24.720494 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.720087    2564 generic.go:356] "Generic (PLEG): container finished" podID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerID="491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7" exitCode=0
Sep 06 10:34:24.720494 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.720149    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"
Sep 06 10:34:24.720494 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.720253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" event={"ID":"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16","Type":"ContainerDied","Data":"491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7"}
Sep 06 10:34:24.720494 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.720296    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9" event={"ID":"af6708a5-0ae5-4dc9-a9ab-05e4308b4c16","Type":"ContainerDied","Data":"be3bb7359b1f281dae6b4fe44709ce20f4b8670990576c349b8fc8eeacd01a5d"}
Sep 06 10:34:24.720494 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.720325    2564 scope.go:117] "RemoveContainer" containerID="491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7"
Sep 06 10:34:24.748022 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.747999    2564 scope.go:117] "RemoveContainer" containerID="80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7"
Sep 06 10:34:24.758731 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.758711    2564 scope.go:117] "RemoveContainer" containerID="491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7"
Sep 06 10:34:24.759102 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:34:24.759079    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7\": container with ID starting with 491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7 not found: ID does not exist" containerID="491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7"
Sep 06 10:34:24.759197 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.759111    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7"} err="failed to get container status \"491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7\": rpc error: code = NotFound desc = could not find container \"491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7\": container with ID starting with 491bca72780c91763afe749d27b78587c6b169fde1d782d6fa71bb838d371ab7 not found: ID does not exist"
Sep 06 10:34:24.759197 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.759129    2564 scope.go:117] "RemoveContainer" containerID="80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7"
Sep 06 10:34:24.759482 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:34:24.759450    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7\": container with ID starting with 80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7 not found: ID does not exist" containerID="80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7"
Sep 06 10:34:24.759528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.759486    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7"} err="failed to get container status \"80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7\": rpc error: code = NotFound desc = could not find container \"80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7\": container with ID starting with 80bad36294007406489762aafc2cd7d064638ff42156d2e4f608aad81b8e8bb7 not found: ID does not exist"
Sep 06 10:34:24.760805 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.760589    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"]
Sep 06 10:34:24.763598 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.763580    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ca3c8694-gw-openshift-default-6459f9c7d4-rcwc9"]
Sep 06 10:34:24.957536 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:24.957500    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"]
Sep 06 10:34:24.959008 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:34:24.958893    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c1a055_6a41_497b_bd39_18bfd4001797.slice/crio-3497eaa1fe190b710b417bf32856ce399ecb2a8623413f114a81011758c740ac WatchSource:0}: Error finding container 3497eaa1fe190b710b417bf32856ce399ecb2a8623413f114a81011758c740ac: Status 404 returned error can't find the container with id 3497eaa1fe190b710b417bf32856ce399ecb2a8623413f114a81011758c740ac
Sep 06 10:34:25.728752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:25.728700    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p" event={"ID":"56c1a055-6a41-497b-bd39-18bfd4001797","Type":"ContainerStarted","Data":"f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7"}
Sep 06 10:34:25.729158 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:25.728770    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:25.729158 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:25.728797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p" event={"ID":"56c1a055-6a41-497b-bd39-18bfd4001797","Type":"ContainerStarted","Data":"3497eaa1fe190b710b417bf32856ce399ecb2a8623413f114a81011758c740ac"}
Sep 06 10:34:25.730502 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:25.730476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4" event={"ID":"8080509e-d8a3-4d19-b91b-8c117b5e1e90","Type":"ContainerStarted","Data":"9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092"}
Sep 06 10:34:25.730638 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:25.730555    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:25.746530 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:25.746481    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p" podStartSLOduration=1.746464808 podStartE2EDuration="1.746464808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:34:25.745482623 +0000 UTC m=+754.228982762" watchObservedRunningTime="2026-09-06 10:34:25.746464808 +0000 UTC m=+754.229964901"
Sep 06 10:34:25.763633 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:25.763578    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4" podStartSLOduration=2.352879688 podStartE2EDuration="2.763560072s" totalImagesPullingTime="410.680384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:34:25.762160004 +0000 UTC m=+754.245660134" watchObservedRunningTime="2026-09-06 10:34:25.763560072 +0000 UTC m=+754.247060221"
Sep 06 10:34:26.029457 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:26.029364    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" path="/var/lib/kubelet/pods/af6708a5-0ae5-4dc9-a9ab-05e4308b4c16/volumes"
Sep 06 10:34:31.743005 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:31.742973    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:36.738872 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:36.738837    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:53.190361 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.190321    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"]
Sep 06 10:34:53.190973 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.190956    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy"
Sep 06 10:34:53.191026 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.190976    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy"
Sep 06 10:34:53.191109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.191099    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy"
Sep 06 10:34:53.191145 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.191120    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy"
Sep 06 10:34:53.194275 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.194254    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.197158 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.197131    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5dda3d\"/\"openshift-service-ca.crt\""
Sep 06 10:34:53.198134 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.198112    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5dda3d\"/\"kube-root-ca.crt\""
Sep 06 10:34:53.198276 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.198139    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5dda3d\"/\"autores-model-5dda3d-kserve-self-signed-certs\""
Sep 06 10:34:53.198437 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.198423    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5dda3d\"/\"default-dockercfg-kn928\""
Sep 06 10:34:53.202334 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.202311    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"]
Sep 06 10:34:53.229760 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.229723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-dshm\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.229760 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.229757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tls-certs\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.230068 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.229793    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-home\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.230068 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.229858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-model-cache\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.230068 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.229978    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqczh\" (UniqueName: \"kubernetes.io/projected/e46fb573-0ae8-40d5-ac64-6d71475e7f20-kube-api-access-nqczh\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.230068 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.230043    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tmp-dir\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.331002 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.330955    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqczh\" (UniqueName: \"kubernetes.io/projected/e46fb573-0ae8-40d5-ac64-6d71475e7f20-kube-api-access-nqczh\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.331240 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.331026    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tmp-dir\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.331240 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.331082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-dshm\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.331240 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.331102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tls-certs\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.331240 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.331137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-home\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.331240 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.331165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-model-cache\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.331529 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.331508    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tmp-dir\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.331615 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.331583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-home\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.331671 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.331630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-model-cache\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.333373 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.333353    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-dshm\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.333618 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.333600    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tls-certs\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.338752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.338728    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqczh\" (UniqueName: \"kubernetes.io/projected/e46fb573-0ae8-40d5-ac64-6d71475e7f20-kube-api-access-nqczh\") pod \"autores-model-5dda3d-kserve-d49bbc695-bsld5\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") " pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.506769 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.506673    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:34:53.783074 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.783047    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"]
Sep 06 10:34:53.784603 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:34:53.784568    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode46fb573_0ae8_40d5_ac64_6d71475e7f20.slice/crio-41fb5dafdb4ddcde7bd4eb4ff11d3e2b2005bd5d0c2bc8fc2c69c27219b64954 WatchSource:0}: Error finding container 41fb5dafdb4ddcde7bd4eb4ff11d3e2b2005bd5d0c2bc8fc2c69c27219b64954: Status 404 returned error can't find the container with id 41fb5dafdb4ddcde7bd4eb4ff11d3e2b2005bd5d0c2bc8fc2c69c27219b64954
Sep 06 10:34:53.886911 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.886875    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5" event={"ID":"e46fb573-0ae8-40d5-ac64-6d71475e7f20","Type":"ContainerStarted","Data":"1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4"}
Sep 06 10:34:53.887077 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.886918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5" event={"ID":"e46fb573-0ae8-40d5-ac64-6d71475e7f20","Type":"ContainerStarted","Data":"41fb5dafdb4ddcde7bd4eb4ff11d3e2b2005bd5d0c2bc8fc2c69c27219b64954"}
Sep 06 10:34:53.905007 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:53.904948    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5" podStartSLOduration=0.904926921 podStartE2EDuration="904.926921ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:34:53.904277204 +0000 UTC m=+782.387777338" watchObservedRunningTime="2026-09-06 10:34:53.904926921 +0000 UTC m=+782.388427031"
Sep 06 10:34:54.338418 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.338384    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"]
Sep 06 10:34:54.338899 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.338632    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4" podUID="8080509e-d8a3-4d19-b91b-8c117b5e1e90" containerName="maas-api" containerID="cri-o://9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092" gracePeriod=30
Sep 06 10:34:54.397843 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.397809    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"]
Sep 06 10:34:54.398093 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.398049    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p" podUID="56c1a055-6a41-497b-bd39-18bfd4001797" containerName="payload-processing" containerID="cri-o://f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7" gracePeriod=30
Sep 06 10:34:54.638854 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.638829    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:54.659032 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.659006    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:54.743156 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.743125    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-schrq\" (UniqueName: \"kubernetes.io/projected/56c1a055-6a41-497b-bd39-18bfd4001797-kube-api-access-schrq\") pod \"56c1a055-6a41-497b-bd39-18bfd4001797\" (UID: \"56c1a055-6a41-497b-bd39-18bfd4001797\") "
Sep 06 10:34:54.743336 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.743167    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw5nm\" (UniqueName: \"kubernetes.io/projected/8080509e-d8a3-4d19-b91b-8c117b5e1e90-kube-api-access-rw5nm\") pod \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") "
Sep 06 10:34:54.743336 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.743242    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c1a055-6a41-497b-bd39-18bfd4001797-plugins-config-volume\") pod \"56c1a055-6a41-497b-bd39-18bfd4001797\" (UID: \"56c1a055-6a41-497b-bd39-18bfd4001797\") "
Sep 06 10:34:54.743336 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.743276    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-maas-api-tls\") pod \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") "
Sep 06 10:34:54.743442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.743398    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8080509e-d8a3-4d19-b91b-8c117b5e1e90-openshift-service-ca\") pod \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") "
Sep 06 10:34:54.743524 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.743503    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-metrics-certs\") pod \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\" (UID: \"8080509e-d8a3-4d19-b91b-8c117b5e1e90\") "
Sep 06 10:34:54.743692 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.743665    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c1a055-6a41-497b-bd39-18bfd4001797-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56c1a055-6a41-497b-bd39-18bfd4001797" (UID: "56c1a055-6a41-497b-bd39-18bfd4001797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:34:54.743784 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.743710    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8080509e-d8a3-4d19-b91b-8c117b5e1e90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8080509e-d8a3-4d19-b91b-8c117b5e1e90" (UID: "8080509e-d8a3-4d19-b91b-8c117b5e1e90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:34:54.743899 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.743881    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c1a055-6a41-497b-bd39-18bfd4001797-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:54.743945 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.743903    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8080509e-d8a3-4d19-b91b-8c117b5e1e90-openshift-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:54.745822 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.745791    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8080509e-d8a3-4d19-b91b-8c117b5e1e90" (UID: "8080509e-d8a3-4d19-b91b-8c117b5e1e90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:34:54.745909 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.745849    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8080509e-d8a3-4d19-b91b-8c117b5e1e90" (UID: "8080509e-d8a3-4d19-b91b-8c117b5e1e90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:34:54.745909 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.745863    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8080509e-d8a3-4d19-b91b-8c117b5e1e90-kube-api-access-rw5nm" (OuterVolumeSpecName: "kube-api-access-rw5nm") pod "8080509e-d8a3-4d19-b91b-8c117b5e1e90" (UID: "8080509e-d8a3-4d19-b91b-8c117b5e1e90"). InnerVolumeSpecName "kube-api-access-rw5nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:34:54.745909 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.745851    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c1a055-6a41-497b-bd39-18bfd4001797-kube-api-access-schrq" (OuterVolumeSpecName: "kube-api-access-schrq") pod "56c1a055-6a41-497b-bd39-18bfd4001797" (UID: "56c1a055-6a41-497b-bd39-18bfd4001797"). InnerVolumeSpecName "kube-api-access-schrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:34:54.844393 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.844362    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw5nm\" (UniqueName: \"kubernetes.io/projected/8080509e-d8a3-4d19-b91b-8c117b5e1e90-kube-api-access-rw5nm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:54.844393 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.844390    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-maas-api-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:54.844573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.844404    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8080509e-d8a3-4d19-b91b-8c117b5e1e90-metrics-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:54.844573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.844417    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-schrq\" (UniqueName: \"kubernetes.io/projected/56c1a055-6a41-497b-bd39-18bfd4001797-kube-api-access-schrq\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:34:54.892908 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.892826    2564 generic.go:356] "Generic (PLEG): container finished" podID="56c1a055-6a41-497b-bd39-18bfd4001797" containerID="f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7" exitCode=0
Sep 06 10:34:54.892908 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.892880    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"
Sep 06 10:34:54.893104 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.892922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p" event={"ID":"56c1a055-6a41-497b-bd39-18bfd4001797","Type":"ContainerDied","Data":"f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7"}
Sep 06 10:34:54.893104 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.892972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p" event={"ID":"56c1a055-6a41-497b-bd39-18bfd4001797","Type":"ContainerDied","Data":"3497eaa1fe190b710b417bf32856ce399ecb2a8623413f114a81011758c740ac"}
Sep 06 10:34:54.893104 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.892991    2564 scope.go:117] "RemoveContainer" containerID="f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7"
Sep 06 10:34:54.894386 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.894361    2564 generic.go:356] "Generic (PLEG): container finished" podID="8080509e-d8a3-4d19-b91b-8c117b5e1e90" containerID="9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092" exitCode=0
Sep 06 10:34:54.894495 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.894386    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4" event={"ID":"8080509e-d8a3-4d19-b91b-8c117b5e1e90","Type":"ContainerDied","Data":"9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092"}
Sep 06 10:34:54.894495 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.894409    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"
Sep 06 10:34:54.894495 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.894431    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4" event={"ID":"8080509e-d8a3-4d19-b91b-8c117b5e1e90","Type":"ContainerDied","Data":"ff64d3a4f2c55022e207c8953f1faab1e4067b7dcb6e0405d826b1390f1dde66"}
Sep 06 10:34:54.902834 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.902813    2564 scope.go:117] "RemoveContainer" containerID="f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7"
Sep 06 10:34:54.903142 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:34:54.903124    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7\": container with ID starting with f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7 not found: ID does not exist" containerID="f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7"
Sep 06 10:34:54.903202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.903150    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7"} err="failed to get container status \"f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7\": rpc error: code = NotFound desc = could not find container \"f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7\": container with ID starting with f91af34208b2b84a782a777d9779c859fde5328f3b139291e750b4d9659634a7 not found: ID does not exist"
Sep 06 10:34:54.903202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.903165    2564 scope.go:117] "RemoveContainer" containerID="9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092"
Sep 06 10:34:54.911060 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.911047    2564 scope.go:117] "RemoveContainer" containerID="9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092"
Sep 06 10:34:54.911337 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:34:54.911320    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092\": container with ID starting with 9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092 not found: ID does not exist" containerID="9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092"
Sep 06 10:34:54.911395 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.911344    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092"} err="failed to get container status \"9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092\": rpc error: code = NotFound desc = could not find container \"9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092\": container with ID starting with 9f50a2abced79b02f9ddbdcb3cb16ddbb2f39ac56d4e0bbdfb99de787c4ac092 not found: ID does not exist"
Sep 06 10:34:54.919090 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.919067    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"]
Sep 06 10:34:54.924487 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.924467    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17db6c23-5b76748dbb-dvdw4"]
Sep 06 10:34:54.934761 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.934742    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"]
Sep 06 10:34:54.939241 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:54.939222    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17db6c23-c9d564899-vph8p"]
Sep 06 10:34:56.029136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:56.029102    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56c1a055-6a41-497b-bd39-18bfd4001797" path="/var/lib/kubelet/pods/56c1a055-6a41-497b-bd39-18bfd4001797/volumes"
Sep 06 10:34:56.029522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:56.029514    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8080509e-d8a3-4d19-b91b-8c117b5e1e90" path="/var/lib/kubelet/pods/8080509e-d8a3-4d19-b91b-8c117b5e1e90/volumes"
Sep 06 10:34:59.969906 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.969870    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"]
Sep 06 10:34:59.970525 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.970482    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c1a055-6a41-497b-bd39-18bfd4001797" containerName="payload-processing"
Sep 06 10:34:59.970613 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.970529    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c1a055-6a41-497b-bd39-18bfd4001797" containerName="payload-processing"
Sep 06 10:34:59.970613 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.970550    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy"
Sep 06 10:34:59.970613 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.970559    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="af6708a5-0ae5-4dc9-a9ab-05e4308b4c16" containerName="istio-proxy"
Sep 06 10:34:59.970613 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.970593    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8080509e-d8a3-4d19-b91b-8c117b5e1e90" containerName="maas-api"
Sep 06 10:34:59.970613 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.970603    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8080509e-d8a3-4d19-b91b-8c117b5e1e90" containerName="maas-api"
Sep 06 10:34:59.970874 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.970711    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8080509e-d8a3-4d19-b91b-8c117b5e1e90" containerName="maas-api"
Sep 06 10:34:59.970874 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.970724    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="56c1a055-6a41-497b-bd39-18bfd4001797" containerName="payload-processing"
Sep 06 10:34:59.974717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.974693    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:34:59.977438 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.977414    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-7bb9997d-serving-cert\""
Sep 06 10:34:59.982609 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:34:59.982588    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"]
Sep 06 10:35:00.096100 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.096065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-metrics-certs\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.096304 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.096200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-maas-api-tls\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.096366 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.096292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4081198d-a27d-4cb9-af58-ceff2e6b3569-openshift-service-ca\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.096451 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.096390    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztlc\" (UniqueName: \"kubernetes.io/projected/4081198d-a27d-4cb9-af58-ceff2e6b3569-kube-api-access-cztlc\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.197749 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.197711    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-maas-api-tls\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.197926 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.197759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4081198d-a27d-4cb9-af58-ceff2e6b3569-openshift-service-ca\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.197926 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.197810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztlc\" (UniqueName: \"kubernetes.io/projected/4081198d-a27d-4cb9-af58-ceff2e6b3569-kube-api-access-cztlc\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.197926 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:35:00.197864    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-serving-cert: secret "maas-api-e2e-derive-7bb9997d-serving-cert" not found
Sep 06 10:35:00.197926 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.197886    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-metrics-certs\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.198158 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:35:00.197942    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-maas-api-tls podName:4081198d-a27d-4cb9-af58-ceff2e6b3569 nodeName:}" failed. No retries permitted until 2026-09-06 10:35:00.697918655 +0000 UTC m=+789.181418757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-maas-api-tls") pod "maas-api-e2e-derive-7bb9997d-55dc597468-hxgms" (UID: "4081198d-a27d-4cb9-af58-ceff2e6b3569") : secret "maas-api-e2e-derive-7bb9997d-serving-cert" not found
Sep 06 10:35:00.198458 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.198437    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4081198d-a27d-4cb9-af58-ceff2e6b3569-openshift-service-ca\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.200327 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.200301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-metrics-certs\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.206381 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.206362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztlc\" (UniqueName: \"kubernetes.io/projected/4081198d-a27d-4cb9-af58-ceff2e6b3569-kube-api-access-cztlc\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.601446 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.601408    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"]
Sep 06 10:35:00.604457 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.604436    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:00.607202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.607177    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-7bb9997d\""
Sep 06 10:35:00.607443 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.607428    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-7bb9997d-dockercfg-jzdxm\""
Sep 06 10:35:00.615834 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.615808    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"]
Sep 06 10:35:00.710357 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.710320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-maas-api-tls\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.710510 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.710365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da43d100-3bdd-4ac7-97d7-e79083db27e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z\" (UID: \"da43d100-3bdd-4ac7-97d7-e79083db27e8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:00.710510 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.710439    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv5bw\" (UniqueName: \"kubernetes.io/projected/da43d100-3bdd-4ac7-97d7-e79083db27e8-kube-api-access-gv5bw\") pod \"payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z\" (UID: \"da43d100-3bdd-4ac7-97d7-e79083db27e8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:00.712917 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.712892    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-maas-api-tls\") pod \"maas-api-e2e-derive-7bb9997d-55dc597468-hxgms\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.811984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.811944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv5bw\" (UniqueName: \"kubernetes.io/projected/da43d100-3bdd-4ac7-97d7-e79083db27e8-kube-api-access-gv5bw\") pod \"payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z\" (UID: \"da43d100-3bdd-4ac7-97d7-e79083db27e8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:00.812153 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.812047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da43d100-3bdd-4ac7-97d7-e79083db27e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z\" (UID: \"da43d100-3bdd-4ac7-97d7-e79083db27e8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:00.812727 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.812706    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da43d100-3bdd-4ac7-97d7-e79083db27e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z\" (UID: \"da43d100-3bdd-4ac7-97d7-e79083db27e8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:00.822555 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.822531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv5bw\" (UniqueName: \"kubernetes.io/projected/da43d100-3bdd-4ac7-97d7-e79083db27e8-kube-api-access-gv5bw\") pod \"payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z\" (UID: \"da43d100-3bdd-4ac7-97d7-e79083db27e8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:00.889543 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.889461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:00.915611 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:00.915584    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:01.166270 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:01.166244    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"]
Sep 06 10:35:01.168522 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:35:01.168493    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4081198d_a27d_4cb9_af58_ceff2e6b3569.slice/crio-a533366d55bb2669c82129175112355a5a0874cb8426c6221c3c4e385a82bfcf WatchSource:0}: Error finding container a533366d55bb2669c82129175112355a5a0874cb8426c6221c3c4e385a82bfcf: Status 404 returned error can't find the container with id a533366d55bb2669c82129175112355a5a0874cb8426c6221c3c4e385a82bfcf
Sep 06 10:35:01.193723 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:01.193700    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"]
Sep 06 10:35:01.195357 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:35:01.195333    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda43d100_3bdd_4ac7_97d7_e79083db27e8.slice/crio-d52f6d2324e5c372c8b7634a0b296ef8e9ea1376b8fd469aefde60fc5fdf27a3 WatchSource:0}: Error finding container d52f6d2324e5c372c8b7634a0b296ef8e9ea1376b8fd469aefde60fc5fdf27a3: Status 404 returned error can't find the container with id d52f6d2324e5c372c8b7634a0b296ef8e9ea1376b8fd469aefde60fc5fdf27a3
Sep 06 10:35:01.936328 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:01.936083    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z" event={"ID":"da43d100-3bdd-4ac7-97d7-e79083db27e8","Type":"ContainerStarted","Data":"613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26"}
Sep 06 10:35:01.936328 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:01.936156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z" event={"ID":"da43d100-3bdd-4ac7-97d7-e79083db27e8","Type":"ContainerStarted","Data":"d52f6d2324e5c372c8b7634a0b296ef8e9ea1376b8fd469aefde60fc5fdf27a3"}
Sep 06 10:35:01.936328 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:01.936203    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:01.938068 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:01.938023    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms" event={"ID":"4081198d-a27d-4cb9-af58-ceff2e6b3569","Type":"ContainerStarted","Data":"c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d"}
Sep 06 10:35:01.938068 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:01.938065    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms" event={"ID":"4081198d-a27d-4cb9-af58-ceff2e6b3569","Type":"ContainerStarted","Data":"a533366d55bb2669c82129175112355a5a0874cb8426c6221c3c4e385a82bfcf"}
Sep 06 10:35:01.938241 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:01.938103    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:01.956333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:01.956287    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z" podStartSLOduration=1.956271009 podStartE2EDuration="1.956271009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:35:01.954245468 +0000 UTC m=+790.437745603" watchObservedRunningTime="2026-09-06 10:35:01.956271009 +0000 UTC m=+790.439771172"
Sep 06 10:35:01.971958 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:01.971909    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms" podStartSLOduration=2.476302756 podStartE2EDuration="2.971892919s" totalImagesPullingTime="495.590163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:35:01.971109042 +0000 UTC m=+790.454609172" watchObservedRunningTime="2026-09-06 10:35:01.971892919 +0000 UTC m=+790.455393048"
Sep 06 10:35:03.507156 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:03.507118    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:35:03.507658 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:03.507166    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:35:03.520591 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:03.520565    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:35:03.960436 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:03.960401    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:35:07.948890 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:07.948857    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:12.945684 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:12.945655    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:17.395602 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.395514    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"]
Sep 06 10:35:17.399505 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.399462    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.403139 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.403116    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04526a\"/\"openshift-service-ca.crt\""
Sep 06 10:35:17.403427 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.403404    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04526a\"/\"test-model-04526a-kserve-self-signed-certs\""
Sep 06 10:35:17.403545 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.403461    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04526a\"/\"default-dockercfg-7dpms\""
Sep 06 10:35:17.403621 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.403405    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04526a\"/\"kube-root-ca.crt\""
Sep 06 10:35:17.408400 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.408029    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"]
Sep 06 10:35:17.477783 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.477750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-model-cache\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.477958 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.477791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-dshm\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.477958 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.477810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzbht\" (UniqueName: \"kubernetes.io/projected/c9a002a7-b923-4784-a4eb-08e981f011fd-kube-api-access-tzbht\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.477958 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.477921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a002a7-b923-4784-a4eb-08e981f011fd-tls-certs\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.478109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.477967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-home\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.478109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.478061    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-tmp-dir\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.578744 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.578701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-tmp-dir\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.578929 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.578807    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-model-cache\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.578929 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.578847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-dshm\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.578929 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.578874    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzbht\" (UniqueName: \"kubernetes.io/projected/c9a002a7-b923-4784-a4eb-08e981f011fd-kube-api-access-tzbht\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.578929 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.578924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a002a7-b923-4784-a4eb-08e981f011fd-tls-certs\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.579187 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.578966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-home\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.579285 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.579185    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-tmp-dir\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.579285 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.579236    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-model-cache\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.579402 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.579327    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-home\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.581142 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.581112    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-dshm\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.581722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.581700    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a002a7-b923-4784-a4eb-08e981f011fd-tls-certs\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.588531 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.588502    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzbht\" (UniqueName: \"kubernetes.io/projected/c9a002a7-b923-4784-a4eb-08e981f011fd-kube-api-access-tzbht\") pod \"test-model-04526a-kserve-9ddcfb9df-dxrxj\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") " pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.711508 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.711410    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:17.984203 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:17.984169    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"]
Sep 06 10:35:17.985981 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:35:17.985953    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9a002a7_b923_4784_a4eb_08e981f011fd.slice/crio-8c241224ad50edf81088a195b54a9a79180da6b10f98cf98acfd4a0c224de317 WatchSource:0}: Error finding container 8c241224ad50edf81088a195b54a9a79180da6b10f98cf98acfd4a0c224de317: Status 404 returned error can't find the container with id 8c241224ad50edf81088a195b54a9a79180da6b10f98cf98acfd4a0c224de317
Sep 06 10:35:18.019972 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:18.019939    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj" event={"ID":"c9a002a7-b923-4784-a4eb-08e981f011fd","Type":"ContainerStarted","Data":"8c241224ad50edf81088a195b54a9a79180da6b10f98cf98acfd4a0c224de317"}
Sep 06 10:35:19.026633 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:19.026596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj" event={"ID":"c9a002a7-b923-4784-a4eb-08e981f011fd","Type":"ContainerStarted","Data":"d44ec3edf1222749d100fa06c31b5e5dace6cb09c48811c7968bcf1b19b87393"}
Sep 06 10:35:19.045019 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:19.044968    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj" podStartSLOduration=2.044952219 podStartE2EDuration="2.044952219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:35:19.042976744 +0000 UTC m=+807.526476851" watchObservedRunningTime="2026-09-06 10:35:19.044952219 +0000 UTC m=+807.528452358"
Sep 06 10:35:19.387718 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:19.387685    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"]
Sep 06 10:35:21.842596 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:21.842562    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"]
Sep 06 10:35:27.712324 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:27.712284    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:27.712324 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:27.712335    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:27.724729 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:27.724700    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:28.084522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:28.084481    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:35:30.417548 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.417506    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"]
Sep 06 10:35:30.417998 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.417770    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms" podUID="4081198d-a27d-4cb9-af58-ceff2e6b3569" containerName="maas-api" containerID="cri-o://c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d" gracePeriod=30
Sep 06 10:35:30.478985 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.478942    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"]
Sep 06 10:35:30.479321 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.479265    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z" podUID="da43d100-3bdd-4ac7-97d7-e79083db27e8" containerName="payload-pre-processing" containerID="cri-o://613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26" gracePeriod=30
Sep 06 10:35:30.725875 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.725850    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:30.764480 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.764456    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:30.815334 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.815295    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-maas-api-tls\") pod \"4081198d-a27d-4cb9-af58-ceff2e6b3569\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") "
Sep 06 10:35:30.815544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.815360    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-metrics-certs\") pod \"4081198d-a27d-4cb9-af58-ceff2e6b3569\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") "
Sep 06 10:35:30.815544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.815395    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4081198d-a27d-4cb9-af58-ceff2e6b3569-openshift-service-ca\") pod \"4081198d-a27d-4cb9-af58-ceff2e6b3569\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") "
Sep 06 10:35:30.815544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.815520    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv5bw\" (UniqueName: \"kubernetes.io/projected/da43d100-3bdd-4ac7-97d7-e79083db27e8-kube-api-access-gv5bw\") pod \"da43d100-3bdd-4ac7-97d7-e79083db27e8\" (UID: \"da43d100-3bdd-4ac7-97d7-e79083db27e8\") "
Sep 06 10:35:30.815719 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.815614    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cztlc\" (UniqueName: \"kubernetes.io/projected/4081198d-a27d-4cb9-af58-ceff2e6b3569-kube-api-access-cztlc\") pod \"4081198d-a27d-4cb9-af58-ceff2e6b3569\" (UID: \"4081198d-a27d-4cb9-af58-ceff2e6b3569\") "
Sep 06 10:35:30.815719 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.815651    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da43d100-3bdd-4ac7-97d7-e79083db27e8-plugins-config-volume\") pod \"da43d100-3bdd-4ac7-97d7-e79083db27e8\" (UID: \"da43d100-3bdd-4ac7-97d7-e79083db27e8\") "
Sep 06 10:35:30.815850 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.815756    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4081198d-a27d-4cb9-af58-ceff2e6b3569-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4081198d-a27d-4cb9-af58-ceff2e6b3569" (UID: "4081198d-a27d-4cb9-af58-ceff2e6b3569"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:35:30.816071 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.816037    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4081198d-a27d-4cb9-af58-ceff2e6b3569-openshift-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:30.816204 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.816077    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da43d100-3bdd-4ac7-97d7-e79083db27e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da43d100-3bdd-4ac7-97d7-e79083db27e8" (UID: "da43d100-3bdd-4ac7-97d7-e79083db27e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:35:30.817631 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.817610    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4081198d-a27d-4cb9-af58-ceff2e6b3569" (UID: "4081198d-a27d-4cb9-af58-ceff2e6b3569"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:35:30.817717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.817659    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4081198d-a27d-4cb9-af58-ceff2e6b3569-kube-api-access-cztlc" (OuterVolumeSpecName: "kube-api-access-cztlc") pod "4081198d-a27d-4cb9-af58-ceff2e6b3569" (UID: "4081198d-a27d-4cb9-af58-ceff2e6b3569"). InnerVolumeSpecName "kube-api-access-cztlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:35:30.817768 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.817704    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da43d100-3bdd-4ac7-97d7-e79083db27e8-kube-api-access-gv5bw" (OuterVolumeSpecName: "kube-api-access-gv5bw") pod "da43d100-3bdd-4ac7-97d7-e79083db27e8" (UID: "da43d100-3bdd-4ac7-97d7-e79083db27e8"). InnerVolumeSpecName "kube-api-access-gv5bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:35:30.817824 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.817804    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4081198d-a27d-4cb9-af58-ceff2e6b3569" (UID: "4081198d-a27d-4cb9-af58-ceff2e6b3569"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:35:30.917071 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.917028    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-metrics-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:30.917071 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.917064    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv5bw\" (UniqueName: \"kubernetes.io/projected/da43d100-3bdd-4ac7-97d7-e79083db27e8-kube-api-access-gv5bw\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:30.917071 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.917073    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cztlc\" (UniqueName: \"kubernetes.io/projected/4081198d-a27d-4cb9-af58-ceff2e6b3569-kube-api-access-cztlc\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:30.917071 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.917083    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da43d100-3bdd-4ac7-97d7-e79083db27e8-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:30.917401 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:30.917092    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4081198d-a27d-4cb9-af58-ceff2e6b3569-maas-api-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:31.089558 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.089526    2564 generic.go:356] "Generic (PLEG): container finished" podID="4081198d-a27d-4cb9-af58-ceff2e6b3569" containerID="c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d" exitCode=0
Sep 06 10:35:31.089741 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.089575    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"
Sep 06 10:35:31.089741 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.089613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms" event={"ID":"4081198d-a27d-4cb9-af58-ceff2e6b3569","Type":"ContainerDied","Data":"c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d"}
Sep 06 10:35:31.089741 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.089666    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms" event={"ID":"4081198d-a27d-4cb9-af58-ceff2e6b3569","Type":"ContainerDied","Data":"a533366d55bb2669c82129175112355a5a0874cb8426c6221c3c4e385a82bfcf"}
Sep 06 10:35:31.089741 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.089696    2564 scope.go:117] "RemoveContainer" containerID="c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d"
Sep 06 10:35:31.091030 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.090988    2564 generic.go:356] "Generic (PLEG): container finished" podID="da43d100-3bdd-4ac7-97d7-e79083db27e8" containerID="613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26" exitCode=0
Sep 06 10:35:31.091030 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.091023    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"
Sep 06 10:35:31.091253 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.091073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z" event={"ID":"da43d100-3bdd-4ac7-97d7-e79083db27e8","Type":"ContainerDied","Data":"613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26"}
Sep 06 10:35:31.091253 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.091119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z" event={"ID":"da43d100-3bdd-4ac7-97d7-e79083db27e8","Type":"ContainerDied","Data":"d52f6d2324e5c372c8b7634a0b296ef8e9ea1376b8fd469aefde60fc5fdf27a3"}
Sep 06 10:35:31.100529 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.100511    2564 scope.go:117] "RemoveContainer" containerID="c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d"
Sep 06 10:35:31.100844 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:35:31.100822    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d\": container with ID starting with c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d not found: ID does not exist" containerID="c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d"
Sep 06 10:35:31.100943 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.100851    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d"} err="failed to get container status \"c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d\": rpc error: code = NotFound desc = could not find container \"c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d\": container with ID starting with c7944e923f6dbc500dd6eabcf516a61c7fb41c5b43e1d050d9ea23f8b06e228d not found: ID does not exist"
Sep 06 10:35:31.100943 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.100870    2564 scope.go:117] "RemoveContainer" containerID="613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26"
Sep 06 10:35:31.109017 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.108998    2564 scope.go:117] "RemoveContainer" containerID="613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26"
Sep 06 10:35:31.109331 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:35:31.109307    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26\": container with ID starting with 613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26 not found: ID does not exist" containerID="613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26"
Sep 06 10:35:31.109389 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.109339    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26"} err="failed to get container status \"613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26\": rpc error: code = NotFound desc = could not find container \"613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26\": container with ID starting with 613fcd1516b21f0c3a8c92485fe47a4e75165882ff0c4d2267b4c67ec9509c26 not found: ID does not exist"
Sep 06 10:35:31.112645 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.112626    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"]
Sep 06 10:35:31.116989 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.116967    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7bb9997d-55dc597468-hxgms"]
Sep 06 10:35:31.127584 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.127563    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"]
Sep 06 10:35:31.131477 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:31.131457    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7bb9997d-65498d694-shs5z"]
Sep 06 10:35:32.029396 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:32.029363    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4081198d-a27d-4cb9-af58-ceff2e6b3569" path="/var/lib/kubelet/pods/4081198d-a27d-4cb9-af58-ceff2e6b3569/volumes"
Sep 06 10:35:32.029818 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:32.029802    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da43d100-3bdd-4ac7-97d7-e79083db27e8" path="/var/lib/kubelet/pods/da43d100-3bdd-4ac7-97d7-e79083db27e8/volumes"
Sep 06 10:35:32.903116 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:32.903082    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"]
Sep 06 10:35:32.903353 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:32.903324    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp" podUID="9324d6f4-6547-435b-b966-94fe1e6fc6ad" containerName="payload-processing" containerID="cri-o://134aa8103ce9edbd4ffc44bd3cbe51085998982a1cd524fad514c1df96e78e6e" gracePeriod=30
Sep 06 10:35:33.105021 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:33.104989    2564 generic.go:356] "Generic (PLEG): container finished" podID="9324d6f4-6547-435b-b966-94fe1e6fc6ad" containerID="134aa8103ce9edbd4ffc44bd3cbe51085998982a1cd524fad514c1df96e78e6e" exitCode=0
Sep 06 10:35:33.105380 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:33.105071    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp" event={"ID":"9324d6f4-6547-435b-b966-94fe1e6fc6ad","Type":"ContainerDied","Data":"134aa8103ce9edbd4ffc44bd3cbe51085998982a1cd524fad514c1df96e78e6e"}
Sep 06 10:35:33.157180 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:33.157123    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:35:33.241486 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:33.241455    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9324d6f4-6547-435b-b966-94fe1e6fc6ad-plugins-config-volume\") pod \"9324d6f4-6547-435b-b966-94fe1e6fc6ad\" (UID: \"9324d6f4-6547-435b-b966-94fe1e6fc6ad\") "
Sep 06 10:35:33.241645 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:33.241535    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxjhm\" (UniqueName: \"kubernetes.io/projected/9324d6f4-6547-435b-b966-94fe1e6fc6ad-kube-api-access-jxjhm\") pod \"9324d6f4-6547-435b-b966-94fe1e6fc6ad\" (UID: \"9324d6f4-6547-435b-b966-94fe1e6fc6ad\") "
Sep 06 10:35:33.241805 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:33.241785    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9324d6f4-6547-435b-b966-94fe1e6fc6ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9324d6f4-6547-435b-b966-94fe1e6fc6ad" (UID: "9324d6f4-6547-435b-b966-94fe1e6fc6ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:35:33.243600 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:33.243570    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9324d6f4-6547-435b-b966-94fe1e6fc6ad-kube-api-access-jxjhm" (OuterVolumeSpecName: "kube-api-access-jxjhm") pod "9324d6f4-6547-435b-b966-94fe1e6fc6ad" (UID: "9324d6f4-6547-435b-b966-94fe1e6fc6ad"). InnerVolumeSpecName "kube-api-access-jxjhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:35:33.343444 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:33.343409    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9324d6f4-6547-435b-b966-94fe1e6fc6ad-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:33.343444 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:33.343439    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxjhm\" (UniqueName: \"kubernetes.io/projected/9324d6f4-6547-435b-b966-94fe1e6fc6ad-kube-api-access-jxjhm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:34.111773 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.111735    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp" event={"ID":"9324d6f4-6547-435b-b966-94fe1e6fc6ad","Type":"ContainerDied","Data":"943c9fd14a24e753b024aa0e614a56868eb631cfe20f0250ff04ab05988d9e3d"}
Sep 06 10:35:34.111773 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.111764    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"
Sep 06 10:35:34.112255 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.111794    2564 scope.go:117] "RemoveContainer" containerID="134aa8103ce9edbd4ffc44bd3cbe51085998982a1cd524fad514c1df96e78e6e"
Sep 06 10:35:34.129636 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.129610    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"]
Sep 06 10:35:34.135671 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.135649    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5dda3d-845577577f-42fdp"]
Sep 06 10:35:34.409725 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.409627    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg" podUID="5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" containerName="main" containerID="cri-o://b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481" gracePeriod=45
Sep 06 10:35:34.668201 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.668131    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:35:34.757002 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.756968    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-model-cache\") pod \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") "
Sep 06 10:35:34.757170 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757032    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-home\") pod \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") "
Sep 06 10:35:34.757170 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757075    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-dshm\") pod \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") "
Sep 06 10:35:34.757170 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757104    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tmp-dir\") pod \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") "
Sep 06 10:35:34.757365 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757171    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swhbb\" (UniqueName: \"kubernetes.io/projected/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-kube-api-access-swhbb\") pod \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") "
Sep 06 10:35:34.757365 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757255    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tls-certs\") pod \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\" (UID: \"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9\") "
Sep 06 10:35:34.757365 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757293    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" (UID: "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:34.757365 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757324    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-home" (OuterVolumeSpecName: "home") pod "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" (UID: "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:34.757558 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757418    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" (UID: "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:34.757673 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757655    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:34.757728 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757677    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:34.757728 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.757715    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:34.759391 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.759372    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-kube-api-access-swhbb" (OuterVolumeSpecName: "kube-api-access-swhbb") pod "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" (UID: "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9"). InnerVolumeSpecName "kube-api-access-swhbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:35:34.759447 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.759403    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" (UID: "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:35:34.759447 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.759440    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-dshm" (OuterVolumeSpecName: "dshm") pod "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" (UID: "5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:34.858616 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.858583    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:34.858616 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.858609    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:34.858616 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:34.858618    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swhbb\" (UniqueName: \"kubernetes.io/projected/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9-kube-api-access-swhbb\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:35.118112 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:35.118076    2564 generic.go:356] "Generic (PLEG): container finished" podID="5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" containerID="b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481" exitCode=0
Sep 06 10:35:35.118624 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:35.118125    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"
Sep 06 10:35:35.118624 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:35.118156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg" event={"ID":"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9","Type":"ContainerDied","Data":"b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481"}
Sep 06 10:35:35.118624 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:35.118193    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg" event={"ID":"5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9","Type":"ContainerDied","Data":"9b59e960def2f56ac2f38e1130539f7c9a31e8502a7cf8c6b3f3981259175428"}
Sep 06 10:35:35.118624 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:35.118206    2564 scope.go:117] "RemoveContainer" containerID="b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481"
Sep 06 10:35:35.127087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:35.127071    2564 scope.go:117] "RemoveContainer" containerID="b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481"
Sep 06 10:35:35.127362 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:35:35.127345    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481\": container with ID starting with b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481 not found: ID does not exist" containerID="b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481"
Sep 06 10:35:35.127415 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:35.127371    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481"} err="failed to get container status \"b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481\": rpc error: code = NotFound desc = could not find container \"b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481\": container with ID starting with b8df2371406d007f5e0bdbb425fdb27e53cc669408df438f41a32638568ec481 not found: ID does not exist"
Sep 06 10:35:35.139606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:35.139583    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"]
Sep 06 10:35:35.143474 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:35.143454    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-30549304-kserve-7c7f44c695-xhkcg"]
Sep 06 10:35:36.028762 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:36.028728    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" path="/var/lib/kubelet/pods/5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9/volumes"
Sep 06 10:35:36.029111 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:36.029098    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9324d6f4-6547-435b-b966-94fe1e6fc6ad" path="/var/lib/kubelet/pods/9324d6f4-6547-435b-b966-94fe1e6fc6ad/volumes"
Sep 06 10:35:36.862283 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:36.862228    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5" podUID="e46fb573-0ae8-40d5-ac64-6d71475e7f20" containerName="main" containerID="cri-o://1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4" gracePeriod=45
Sep 06 10:35:37.111030 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.111000    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:35:37.132645 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.132571    2564 generic.go:356] "Generic (PLEG): container finished" podID="e46fb573-0ae8-40d5-ac64-6d71475e7f20" containerID="1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4" exitCode=0
Sep 06 10:35:37.132645 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.132611    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"
Sep 06 10:35:37.132645 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.132628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5" event={"ID":"e46fb573-0ae8-40d5-ac64-6d71475e7f20","Type":"ContainerDied","Data":"1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4"}
Sep 06 10:35:37.132645 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.132656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5" event={"ID":"e46fb573-0ae8-40d5-ac64-6d71475e7f20","Type":"ContainerDied","Data":"41fb5dafdb4ddcde7bd4eb4ff11d3e2b2005bd5d0c2bc8fc2c69c27219b64954"}
Sep 06 10:35:37.132967 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.132673    2564 scope.go:117] "RemoveContainer" containerID="1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4"
Sep 06 10:35:37.141427 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.141400    2564 scope.go:117] "RemoveContainer" containerID="1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4"
Sep 06 10:35:37.141685 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:35:37.141668    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4\": container with ID starting with 1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4 not found: ID does not exist" containerID="1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4"
Sep 06 10:35:37.141737 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.141692    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4"} err="failed to get container status \"1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4\": rpc error: code = NotFound desc = could not find container \"1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4\": container with ID starting with 1feef5233cf5a5a308efd8bea01d6d1c0695e1ffcaeab0e869f34f3215dad0b4 not found: ID does not exist"
Sep 06 10:35:37.181066 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.181035    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tls-certs\") pod \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") "
Sep 06 10:35:37.181066 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.181070    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqczh\" (UniqueName: \"kubernetes.io/projected/e46fb573-0ae8-40d5-ac64-6d71475e7f20-kube-api-access-nqczh\") pod \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") "
Sep 06 10:35:37.181285 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.181119    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-home\") pod \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") "
Sep 06 10:35:37.181285 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.181159    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-model-cache\") pod \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") "
Sep 06 10:35:37.181285 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.181204    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tmp-dir\") pod \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") "
Sep 06 10:35:37.181285 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.181260    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-dshm\") pod \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\" (UID: \"e46fb573-0ae8-40d5-ac64-6d71475e7f20\") "
Sep 06 10:35:37.181493 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.181462    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-home" (OuterVolumeSpecName: "home") pod "e46fb573-0ae8-40d5-ac64-6d71475e7f20" (UID: "e46fb573-0ae8-40d5-ac64-6d71475e7f20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:37.181546 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.181479    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-model-cache" (OuterVolumeSpecName: "model-cache") pod "e46fb573-0ae8-40d5-ac64-6d71475e7f20" (UID: "e46fb573-0ae8-40d5-ac64-6d71475e7f20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:37.181546 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.181533    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e46fb573-0ae8-40d5-ac64-6d71475e7f20" (UID: "e46fb573-0ae8-40d5-ac64-6d71475e7f20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:37.182341 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.182270    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:37.182341 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.182294    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:37.182341 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.182308    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:37.183590 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.183564    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46fb573-0ae8-40d5-ac64-6d71475e7f20-kube-api-access-nqczh" (OuterVolumeSpecName: "kube-api-access-nqczh") pod "e46fb573-0ae8-40d5-ac64-6d71475e7f20" (UID: "e46fb573-0ae8-40d5-ac64-6d71475e7f20"). InnerVolumeSpecName "kube-api-access-nqczh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:35:37.183761 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.183740    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e46fb573-0ae8-40d5-ac64-6d71475e7f20" (UID: "e46fb573-0ae8-40d5-ac64-6d71475e7f20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:35:37.184039 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.184019    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-dshm" (OuterVolumeSpecName: "dshm") pod "e46fb573-0ae8-40d5-ac64-6d71475e7f20" (UID: "e46fb573-0ae8-40d5-ac64-6d71475e7f20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:37.283670 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.283636    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46fb573-0ae8-40d5-ac64-6d71475e7f20-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:37.283670 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.283667    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqczh\" (UniqueName: \"kubernetes.io/projected/e46fb573-0ae8-40d5-ac64-6d71475e7f20-kube-api-access-nqczh\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:37.283670 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.283677    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46fb573-0ae8-40d5-ac64-6d71475e7f20-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:37.453763 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.453731    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"]
Sep 06 10:35:37.461108 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:37.461072    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-5dda3d/autores-model-5dda3d-kserve-d49bbc695-bsld5"]
Sep 06 10:35:38.028627 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:38.028589    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e46fb573-0ae8-40d5-ac64-6d71475e7f20" path="/var/lib/kubelet/pods/e46fb573-0ae8-40d5-ac64-6d71475e7f20/volumes"
Sep 06 10:35:40.233589 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.233546    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"]
Sep 06 10:35:40.234071 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.233845    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy" containerID="cri-o://9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c" gracePeriod=30
Sep 06 10:35:40.989586 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.989547    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"]
Sep 06 10:35:40.990124 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990109    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" containerName="main"
Sep 06 10:35:40.990179 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990129    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" containerName="main"
Sep 06 10:35:40.990179 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990144    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da43d100-3bdd-4ac7-97d7-e79083db27e8" containerName="payload-pre-processing"
Sep 06 10:35:40.990179 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990155    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da43d100-3bdd-4ac7-97d7-e79083db27e8" containerName="payload-pre-processing"
Sep 06 10:35:40.990298 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990180    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9324d6f4-6547-435b-b966-94fe1e6fc6ad" containerName="payload-processing"
Sep 06 10:35:40.990298 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990189    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9324d6f4-6547-435b-b966-94fe1e6fc6ad" containerName="payload-processing"
Sep 06 10:35:40.990298 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990236    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4081198d-a27d-4cb9-af58-ceff2e6b3569" containerName="maas-api"
Sep 06 10:35:40.990298 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990245    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4081198d-a27d-4cb9-af58-ceff2e6b3569" containerName="maas-api"
Sep 06 10:35:40.990298 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990255    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e46fb573-0ae8-40d5-ac64-6d71475e7f20" containerName="main"
Sep 06 10:35:40.990298 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990263    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e46fb573-0ae8-40d5-ac64-6d71475e7f20" containerName="main"
Sep 06 10:35:40.990476 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990347    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e0a10ec-8a7d-41c6-b3f6-ccb6536aa1f9" containerName="main"
Sep 06 10:35:40.990476 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990359    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9324d6f4-6547-435b-b966-94fe1e6fc6ad" containerName="payload-processing"
Sep 06 10:35:40.990476 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990370    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="da43d100-3bdd-4ac7-97d7-e79083db27e8" containerName="payload-pre-processing"
Sep 06 10:35:40.990476 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990381    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e46fb573-0ae8-40d5-ac64-6d71475e7f20" containerName="main"
Sep 06 10:35:40.990476 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.990389    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4081198d-a27d-4cb9-af58-ceff2e6b3569" containerName="maas-api"
Sep 06 10:35:40.995001 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.994978    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:40.997951 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:40.997927    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-6c8f35-openshift-default-dockercfg-ptzz4\""
Sep 06 10:35:41.002542 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.002519    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"]
Sep 06 10:35:41.118691 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.118649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-socket\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.118862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.118716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-certs\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.118862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.118745    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac056971-b231-4655-9543-b5ab8aca8311-istio-podinfo\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.118862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.118763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-data\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.118862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.118778    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac056971-b231-4655-9543-b5ab8aca8311-istiod-ca-cert\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.118862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.118802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgzl\" (UniqueName: \"kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-kube-api-access-zxgzl\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.119095 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.118873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-istio-token\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.119095 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.118922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-credential-socket\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.119095 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.118940    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-envoy\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.219830 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.219792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-certs\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220005 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.219837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac056971-b231-4655-9543-b5ab8aca8311-istio-podinfo\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220005 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.219861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-data\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220005 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.219935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac056971-b231-4655-9543-b5ab8aca8311-istiod-ca-cert\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220005 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.219978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgzl\" (UniqueName: \"kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-kube-api-access-zxgzl\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220268 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.220030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-istio-token\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220268 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.220086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-credential-socket\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220268 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.220117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-envoy\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220268 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.220151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-socket\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220484 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.220266    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-certs\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220484 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.220294    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-data\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220737 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.220709    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-credential-socket\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220737 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.220730    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-socket\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.220890 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.220734    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac056971-b231-4655-9543-b5ab8aca8311-istiod-ca-cert\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.222775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.222753    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-envoy\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.223078 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.223059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac056971-b231-4655-9543-b5ab8aca8311-istio-podinfo\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.228185 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.228165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-istio-token\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.228438 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.228418    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgzl\" (UniqueName: \"kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-kube-api-access-zxgzl\") pod \"e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") " pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.307383 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.307346    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:41.584810 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.584780    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"]
Sep 06 10:35:41.586392 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:35:41.586363    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac056971_b231_4655_9543_b5ab8aca8311.slice/crio-246becc337596fede60ba345de69a18b8a026a5a1abd382335865e9eabcf407c WatchSource:0}: Error finding container 246becc337596fede60ba345de69a18b8a026a5a1abd382335865e9eabcf407c: Status 404 returned error can't find the container with id 246becc337596fede60ba345de69a18b8a026a5a1abd382335865e9eabcf407c
Sep 06 10:35:41.588755 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.588719    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:35:41.588845 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.588804    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:35:41.588890 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:41.588842    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:35:42.160188 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:42.160147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" event={"ID":"ac056971-b231-4655-9543-b5ab8aca8311","Type":"ContainerStarted","Data":"800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31"}
Sep 06 10:35:42.160362 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:42.160230    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" event={"ID":"ac056971-b231-4655-9543-b5ab8aca8311","Type":"ContainerStarted","Data":"246becc337596fede60ba345de69a18b8a026a5a1abd382335865e9eabcf407c"}
Sep 06 10:35:42.181782 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:42.181736    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" podStartSLOduration=2.181721852 podStartE2EDuration="2.181721852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:35:42.18046693 +0000 UTC m=+830.663967050" watchObservedRunningTime="2026-09-06 10:35:42.181721852 +0000 UTC m=+830.665221960"
Sep 06 10:35:42.308558 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:42.308521    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:42.312984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:42.312956    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:43.165809 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:43.165785    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:43.166935 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:43.166919    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:35:44.659132 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:44.659095    2564 patch_prober.go:28] interesting pod/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 06 10:35:44.659524 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:44.659152    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 10:35:45.487651 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.487625    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:35:45.661757 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.661705    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-certs\") pod \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") "
Sep 06 10:35:45.662277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.661783    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-data\") pod \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") "
Sep 06 10:35:45.662277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.661832    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-socket\") pod \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") "
Sep 06 10:35:45.662277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.661882    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-podinfo\") pod \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") "
Sep 06 10:35:45.662277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.661956    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istiod-ca-cert\") pod \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") "
Sep 06 10:35:45.662277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.661980    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-credential-socket\") pod \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") "
Sep 06 10:35:45.662277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662016    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-envoy\") pod \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") "
Sep 06 10:35:45.662277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662054    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-token\") pod \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") "
Sep 06 10:35:45.662277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662098    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfcbj\" (UniqueName: \"kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-kube-api-access-nfcbj\") pod \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\" (UID: \"4104a875-ff2b-432e-af78-1ed71b7fdd7b\") "
Sep 06 10:35:45.662650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662021    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4104a875-ff2b-432e-af78-1ed71b7fdd7b" (UID: "4104a875-ff2b-432e-af78-1ed71b7fdd7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:45.662650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-data" (OuterVolumeSpecName: "istio-data") pod "4104a875-ff2b-432e-af78-1ed71b7fdd7b" (UID: "4104a875-ff2b-432e-af78-1ed71b7fdd7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:45.662650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662245    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4104a875-ff2b-432e-af78-1ed71b7fdd7b" (UID: "4104a875-ff2b-432e-af78-1ed71b7fdd7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:45.662650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662326    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4104a875-ff2b-432e-af78-1ed71b7fdd7b" (UID: "4104a875-ff2b-432e-af78-1ed71b7fdd7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:45.662650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662420    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4104a875-ff2b-432e-af78-1ed71b7fdd7b" (UID: "4104a875-ff2b-432e-af78-1ed71b7fdd7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:35:45.662650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662479    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:45.662650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662500    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-data\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:45.662650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662515    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-workload-socket\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:45.662650 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.662528    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-credential-socket\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:45.664324 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.664284    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4104a875-ff2b-432e-af78-1ed71b7fdd7b" (UID: "4104a875-ff2b-432e-af78-1ed71b7fdd7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 10:35:45.664442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.664365    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-token" (OuterVolumeSpecName: "istio-token") pod "4104a875-ff2b-432e-af78-1ed71b7fdd7b" (UID: "4104a875-ff2b-432e-af78-1ed71b7fdd7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:35:45.664554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.664537    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4104a875-ff2b-432e-af78-1ed71b7fdd7b" (UID: "4104a875-ff2b-432e-af78-1ed71b7fdd7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:45.664602 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.664574    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-kube-api-access-nfcbj" (OuterVolumeSpecName: "kube-api-access-nfcbj") pod "4104a875-ff2b-432e-af78-1ed71b7fdd7b" (UID: "4104a875-ff2b-432e-af78-1ed71b7fdd7b"). InnerVolumeSpecName "kube-api-access-nfcbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:35:45.763187 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.763147    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-podinfo\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:45.763187 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.763179    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istiod-ca-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:45.763187 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.763189    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-envoy\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:45.763187 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.763197    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-istio-token\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:45.763533 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.763205    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfcbj\" (UniqueName: \"kubernetes.io/projected/4104a875-ff2b-432e-af78-1ed71b7fdd7b-kube-api-access-nfcbj\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:35:45.863604 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.863564    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"]
Sep 06 10:35:45.864117 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.864102    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy"
Sep 06 10:35:45.864154 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.864120    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy"
Sep 06 10:35:45.864154 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.864137    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy"
Sep 06 10:35:45.864154 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.864143    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy"
Sep 06 10:35:45.864359 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.864228    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy"
Sep 06 10:35:45.864359 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.864239    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerName="istio-proxy"
Sep 06 10:35:45.867725 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.867702    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:35:45.870547 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.870523    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"]
Sep 06 10:35:45.871026 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.871006    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-6c8f35\""
Sep 06 10:35:45.874071 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.874051    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:35:45.876397 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.876380    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-6c8f35-dockercfg-c7b6f\""
Sep 06 10:35:45.888392 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.888370    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"]
Sep 06 10:35:45.890243 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.890222    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"]
Sep 06 10:35:45.965329 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.965236    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbfq6\" (UniqueName: \"kubernetes.io/projected/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-kube-api-access-xbfq6\") pod \"payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84\" (UID: \"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:35:45.965329 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.965279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84\" (UID: \"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:35:45.965329 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.965306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkmcn\" (UniqueName: \"kubernetes.io/projected/7d080f07-4766-4613-847c-009949551f10-kube-api-access-qkmcn\") pod \"payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb\" (UID: \"7d080f07-4766-4613-847c-009949551f10\") " pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:35:45.965541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:45.965362    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d080f07-4766-4613-847c-009949551f10-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb\" (UID: \"7d080f07-4766-4613-847c-009949551f10\") " pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:35:46.066426 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.066391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbfq6\" (UniqueName: \"kubernetes.io/projected/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-kube-api-access-xbfq6\") pod \"payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84\" (UID: \"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:35:46.066573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.066434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84\" (UID: \"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:35:46.066573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.066547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkmcn\" (UniqueName: \"kubernetes.io/projected/7d080f07-4766-4613-847c-009949551f10-kube-api-access-qkmcn\") pod \"payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb\" (UID: \"7d080f07-4766-4613-847c-009949551f10\") " pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:35:46.066653 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.066593    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d080f07-4766-4613-847c-009949551f10-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb\" (UID: \"7d080f07-4766-4613-847c-009949551f10\") " pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:35:46.067107 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.067090    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84\" (UID: \"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:35:46.067269 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.067251    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d080f07-4766-4613-847c-009949551f10-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb\" (UID: \"7d080f07-4766-4613-847c-009949551f10\") " pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:35:46.085819 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.085788    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbfq6\" (UniqueName: \"kubernetes.io/projected/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-kube-api-access-xbfq6\") pod \"payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84\" (UID: \"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:35:46.086690 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.086670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkmcn\" (UniqueName: \"kubernetes.io/projected/7d080f07-4766-4613-847c-009949551f10-kube-api-access-qkmcn\") pod \"payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb\" (UID: \"7d080f07-4766-4613-847c-009949551f10\") " pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:35:46.180113 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.180084    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:35:46.181772 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.181746    2564 generic.go:356] "Generic (PLEG): container finished" podID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" containerID="9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c" exitCode=0
Sep 06 10:35:46.181892 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.181788    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"
Sep 06 10:35:46.181892 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.181832    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" event={"ID":"4104a875-ff2b-432e-af78-1ed71b7fdd7b","Type":"ContainerDied","Data":"9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c"}
Sep 06 10:35:46.181892 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.181887    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt" event={"ID":"4104a875-ff2b-432e-af78-1ed71b7fdd7b","Type":"ContainerDied","Data":"5e7b45dca0312ab2c3876d35591408b932e65752de2787c33510e34fffe07c2a"}
Sep 06 10:35:46.181996 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.181911    2564 scope.go:117] "RemoveContainer" containerID="9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c"
Sep 06 10:35:46.185989 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.185906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:35:46.196253 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.196231    2564 scope.go:117] "RemoveContainer" containerID="6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318"
Sep 06 10:35:46.203808 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.203778    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"]
Sep 06 10:35:46.206334 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.206314    2564 scope.go:117] "RemoveContainer" containerID="9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c"
Sep 06 10:35:46.206838 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:35:46.206752    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c\": container with ID starting with 9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c not found: ID does not exist" containerID="9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c"
Sep 06 10:35:46.206988 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.206931    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c"} err="failed to get container status \"9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c\": rpc error: code = NotFound desc = could not find container \"9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c\": container with ID starting with 9491d45fb47f7937ce29a08f85cce03071b877e6c4853303b7df071949a3d37c not found: ID does not exist"
Sep 06 10:35:46.207109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.207022    2564 scope.go:117] "RemoveContainer" containerID="6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318"
Sep 06 10:35:46.207655 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:35:46.207490    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318\": container with ID starting with 6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318 not found: ID does not exist" containerID="6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318"
Sep 06 10:35:46.207655 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.207526    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318"} err="failed to get container status \"6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318\": rpc error: code = NotFound desc = could not find container \"6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318\": container with ID starting with 6488b46cd6fcbce58fafc34e216d661f3d0acf8fbd8df98341a1051a7b815318 not found: ID does not exist"
Sep 06 10:35:46.212271 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.212249    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-5dda3d-openshift-default-854c8d96b4-vl7gt"]
Sep 06 10:35:46.462137 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.462111    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"]
Sep 06 10:35:46.464805 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:35:46.464773    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb2ba87_0405_45bf_9e1a_5a06472d7bc3.slice/crio-31eec51822bb3b81f90d9c00ab5a3b71add8cf7c3f895061a67def9bddd31330 WatchSource:0}: Error finding container 31eec51822bb3b81f90d9c00ab5a3b71add8cf7c3f895061a67def9bddd31330: Status 404 returned error can't find the container with id 31eec51822bb3b81f90d9c00ab5a3b71add8cf7c3f895061a67def9bddd31330
Sep 06 10:35:46.466163 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:46.466137    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"]
Sep 06 10:35:46.467966 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:35:46.467943    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d080f07_4766_4613_847c_009949551f10.slice/crio-f49071e8027aaea674e55ed0931a556aa86a8e3393a4a33518c84f30670af366 WatchSource:0}: Error finding container f49071e8027aaea674e55ed0931a556aa86a8e3393a4a33518c84f30670af366: Status 404 returned error can't find the container with id f49071e8027aaea674e55ed0931a556aa86a8e3393a4a33518c84f30670af366
Sep 06 10:35:47.190585 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:47.190545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84" event={"ID":"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3","Type":"ContainerStarted","Data":"7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e"}
Sep 06 10:35:47.191088 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:47.190647    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84" event={"ID":"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3","Type":"ContainerStarted","Data":"31eec51822bb3b81f90d9c00ab5a3b71add8cf7c3f895061a67def9bddd31330"}
Sep 06 10:35:47.191088 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:47.190678    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:35:47.193227 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:47.193174    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb" event={"ID":"7d080f07-4766-4613-847c-009949551f10","Type":"ContainerStarted","Data":"7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199"}
Sep 06 10:35:47.193355 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:47.193244    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb" event={"ID":"7d080f07-4766-4613-847c-009949551f10","Type":"ContainerStarted","Data":"f49071e8027aaea674e55ed0931a556aa86a8e3393a4a33518c84f30670af366"}
Sep 06 10:35:47.193355 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:47.193318    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:35:47.208381 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:47.208334    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84" podStartSLOduration=2.208318647 podStartE2EDuration="2.208318647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:35:47.206251159 +0000 UTC m=+835.689751360" watchObservedRunningTime="2026-09-06 10:35:47.208318647 +0000 UTC m=+835.691818756"
Sep 06 10:35:47.222426 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:47.222364    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb" podStartSLOduration=2.222343427 podStartE2EDuration="2.222343427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:35:47.221986586 +0000 UTC m=+835.705486717" watchObservedRunningTime="2026-09-06 10:35:47.222343427 +0000 UTC m=+835.705843537"
Sep 06 10:35:48.029552 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:48.029517    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4104a875-ff2b-432e-af78-1ed71b7fdd7b" path="/var/lib/kubelet/pods/4104a875-ff2b-432e-af78-1ed71b7fdd7b/volumes"
Sep 06 10:35:58.201139 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:58.201105    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:35:58.201617 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:58.201187    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:35:59.207414 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:59.207378    2564 patch_prober.go:28] interesting pod/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 10:35:59.207856 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:35:59.207452    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 10:36:01.281149 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:01.281115    2564 generic.go:356] "Generic (PLEG): container finished" podID="ac056971-b231-4655-9543-b5ab8aca8311" containerID="800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31" exitCode=0
Sep 06 10:36:01.281588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:01.281179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" event={"ID":"ac056971-b231-4655-9543-b5ab8aca8311","Type":"ContainerDied","Data":"800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31"}
Sep 06 10:36:01.281630 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:01.281587    2564 scope.go:117] "RemoveContainer" containerID="800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31"
Sep 06 10:36:01.282668 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:01.282634    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:36:01.282733 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:01.282717    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:36:01.282770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:01.282753    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:36:02.288405 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:02.288370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" event={"ID":"ac056971-b231-4655-9543-b5ab8aca8311","Type":"ContainerStarted","Data":"15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94"}
Sep 06 10:36:02.308405 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:02.308365    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:36:02.308899 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:02.308870    2564 patch_prober.go:28] interesting pod/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 06 10:36:02.309105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:02.309078    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 06 10:36:03.308345 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:03.308302    2564 patch_prober.go:28] interesting pod/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 06 10:36:03.308858 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:03.308376    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 06 10:36:04.308519 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:04.308477    2564 patch_prober.go:28] interesting pod/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 06 10:36:04.309038 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:04.308554    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 06 10:36:05.308676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:05.308637    2564 patch_prober.go:28] interesting pod/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 06 10:36:05.309293 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:05.309255    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 06 10:36:06.308171 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:06.308129    2564 patch_prober.go:28] interesting pod/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 06 10:36:06.308525 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:06.308482    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 06 10:36:07.308167 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:07.308132    2564 patch_prober.go:28] interesting pod/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 06 10:36:07.308597 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:07.308229    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 06 10:36:08.311487 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:08.311450    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:36:08.311971 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:08.311852    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:36:08.312564 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:08.312543    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:36:17.490140 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.490103    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"]
Sep 06 10:36:17.493865 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.493842    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.496601 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.496579    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bd3407\"/\"kube-root-ca.crt\""
Sep 06 10:36:17.497673 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.497635    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bd3407\"/\"openshift-service-ca.crt\""
Sep 06 10:36:17.497673 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.497653    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bd3407\"/\"default-dockercfg-skchs\""
Sep 06 10:36:17.497673 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.497665    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bd3407\"/\"test-model-bd3407-kserve-self-signed-certs\""
Sep 06 10:36:17.503539 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.503516    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"]
Sep 06 10:36:17.567606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.567568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6kb\" (UniqueName: \"kubernetes.io/projected/3e95ca89-abd5-4489-a938-7d364f08a68f-kube-api-access-9q6kb\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.567788 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.567669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e95ca89-abd5-4489-a938-7d364f08a68f-tls-certs\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.567788 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.567710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-home\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.567788 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.567737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-model-cache\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.567788 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.567754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-tmp-dir\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.567788 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.567777    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-dshm\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.668875 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.668838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e95ca89-abd5-4489-a938-7d364f08a68f-tls-certs\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.668875 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.668880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-home\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.669087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.668900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-model-cache\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.669087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.669024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-tmp-dir\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.669087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.669055    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-dshm\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.669245 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.669151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6kb\" (UniqueName: \"kubernetes.io/projected/3e95ca89-abd5-4489-a938-7d364f08a68f-kube-api-access-9q6kb\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.669321 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.669301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-model-cache\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.669384 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.669347    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-home\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.669441 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.669375    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-tmp-dir\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.671283 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.671254    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-dshm\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.671422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.671405    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e95ca89-abd5-4489-a938-7d364f08a68f-tls-certs\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.679507 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.679482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6kb\" (UniqueName: \"kubernetes.io/projected/3e95ca89-abd5-4489-a938-7d364f08a68f-kube-api-access-9q6kb\") pod \"test-model-bd3407-kserve-57676b4c9c-wglqm\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") " pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:17.805399 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:17.805363    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:18.073879 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:18.073853    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"]
Sep 06 10:36:18.076082 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:36:18.076056    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e95ca89_abd5_4489_a938_7d364f08a68f.slice/crio-0344087bf6cfa0eb2775eaf4b5db6b821ba5b058db0b6cb59cb267fcd5c65962 WatchSource:0}: Error finding container 0344087bf6cfa0eb2775eaf4b5db6b821ba5b058db0b6cb59cb267fcd5c65962: Status 404 returned error can't find the container with id 0344087bf6cfa0eb2775eaf4b5db6b821ba5b058db0b6cb59cb267fcd5c65962
Sep 06 10:36:18.380554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:18.380455    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm" event={"ID":"3e95ca89-abd5-4489-a938-7d364f08a68f","Type":"ContainerStarted","Data":"b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05"}
Sep 06 10:36:18.380554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:18.380512    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm" event={"ID":"3e95ca89-abd5-4489-a938-7d364f08a68f","Type":"ContainerStarted","Data":"0344087bf6cfa0eb2775eaf4b5db6b821ba5b058db0b6cb59cb267fcd5c65962"}
Sep 06 10:36:18.397531 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:18.397468    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm" podStartSLOduration=1.397450592 podStartE2EDuration="1.397450592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:36:18.395926732 +0000 UTC m=+866.879426840" watchObservedRunningTime="2026-09-06 10:36:18.397450592 +0000 UTC m=+866.880950702"
Sep 06 10:36:27.806554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:27.806517    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:27.806963 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:27.806580    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:27.819324 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:27.819299    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:28.441488 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:28.441453    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:36:30.493342 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.493310    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"]
Sep 06 10:36:30.495844 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.495827    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.499341 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.499315    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-6c8f35\"/\"openshift-service-ca.crt\""
Sep 06 10:36:30.499484 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.499339    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-6c8f35\"/\"explicit-model-6c8f35-kserve-self-signed-certs\""
Sep 06 10:36:30.499484 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.499339    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-6c8f35\"/\"default-dockercfg-c6stw\""
Sep 06 10:36:30.499602 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.499484    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-6c8f35\"/\"kube-root-ca.crt\""
Sep 06 10:36:30.504687 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.504652    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"]
Sep 06 10:36:30.596842 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.596806    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-tmp-dir\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.597015 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.596857    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-dshm\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.597015 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.596937    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.597015 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.596985    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvt9r\" (UniqueName: \"kubernetes.io/projected/fcf2da22-b99a-4ea8-b584-4996795c062c-kube-api-access-hvt9r\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.597121 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.597044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-home\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.597121 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.597070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-model-cache\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.698105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.698069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-tmp-dir\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.698308 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.698132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-dshm\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.698308 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.698175    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.698308 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.698232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvt9r\" (UniqueName: \"kubernetes.io/projected/fcf2da22-b99a-4ea8-b584-4996795c062c-kube-api-access-hvt9r\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.698308 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.698270    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-home\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.698308 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.698296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-model-cache\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.698594 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.698502    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-tmp-dir\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.698594 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.698564    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-home\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.698709 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.698669    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-model-cache\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.700429 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.700404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-dshm\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.700663 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.700647    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.705611 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.705586    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvt9r\" (UniqueName: \"kubernetes.io/projected/fcf2da22-b99a-4ea8-b584-4996795c062c-kube-api-access-hvt9r\") pod \"explicit-model-6c8f35-kserve-599c5cf499-4c62d\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") " pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:30.806303 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:30.806274    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:31.074472 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:31.074444    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"]
Sep 06 10:36:31.076466 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:36:31.076417    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcf2da22_b99a_4ea8_b584_4996795c062c.slice/crio-bb95b90eab98fbb50e3727b994a1ffaa0fa1a79fef26a4c29e9e9dfcbe561267 WatchSource:0}: Error finding container bb95b90eab98fbb50e3727b994a1ffaa0fa1a79fef26a4c29e9e9dfcbe561267: Status 404 returned error can't find the container with id bb95b90eab98fbb50e3727b994a1ffaa0fa1a79fef26a4c29e9e9dfcbe561267
Sep 06 10:36:31.445502 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:31.445414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d" event={"ID":"fcf2da22-b99a-4ea8-b584-4996795c062c","Type":"ContainerStarted","Data":"c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76"}
Sep 06 10:36:31.445502 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:31.445458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d" event={"ID":"fcf2da22-b99a-4ea8-b584-4996795c062c","Type":"ContainerStarted","Data":"bb95b90eab98fbb50e3727b994a1ffaa0fa1a79fef26a4c29e9e9dfcbe561267"}
Sep 06 10:36:31.468798 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:31.468748    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d" podStartSLOduration=1.468732275 podStartE2EDuration="1.468732275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:36:31.466702251 +0000 UTC m=+879.950202386" watchObservedRunningTime="2026-09-06 10:36:31.468732275 +0000 UTC m=+879.952232384"
Sep 06 10:36:40.807387 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:40.807353    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:40.807831 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:40.807410    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:40.820132 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:40.820109    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:41.499025 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:41.498996    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"]
Sep 06 10:36:41.512032 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:41.511994    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:41.609332 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:41.609301    2564 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-self-signed-certs: secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:41.609498 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:41.609386    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs podName:fcf2da22-b99a-4ea8-b584-4996795c062c nodeName:}" failed. No retries permitted until 2026-09-06 10:36:42.109365601 +0000 UTC m=+890.592865688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs") pod "explicit-model-6c8f35-kserve-599c5cf499-4c62d" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c") : secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:42.114871 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:42.114783    2564 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-self-signed-certs: secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:42.115358 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:42.114893    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs podName:fcf2da22-b99a-4ea8-b584-4996795c062c nodeName:}" failed. No retries permitted until 2026-09-06 10:36:43.114866279 +0000 UTC m=+891.598366370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs") pod "explicit-model-6c8f35-kserve-599c5cf499-4c62d" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c") : secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:43.124200 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:43.124174    2564 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-self-signed-certs: secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:43.124569 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:43.124248    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs podName:fcf2da22-b99a-4ea8-b584-4996795c062c nodeName:}" failed. No retries permitted until 2026-09-06 10:36:45.124234163 +0000 UTC m=+893.607734250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs") pod "explicit-model-6c8f35-kserve-599c5cf499-4c62d" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c") : secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:45.143728 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:45.143636    2564 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-self-signed-certs: secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:45.143728 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:45.143714    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs podName:fcf2da22-b99a-4ea8-b584-4996795c062c nodeName:}" failed. No retries permitted until 2026-09-06 10:36:49.143700342 +0000 UTC m=+897.627200429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs") pod "explicit-model-6c8f35-kserve-599c5cf499-4c62d" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c") : secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:49.181696 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:49.181663    2564 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-self-signed-certs: secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:49.182097 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:49.181752    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs podName:fcf2da22-b99a-4ea8-b584-4996795c062c nodeName:}" failed. No retries permitted until 2026-09-06 10:36:57.181733923 +0000 UTC m=+905.665234009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs") pod "explicit-model-6c8f35-kserve-599c5cf499-4c62d" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c") : secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:51.978740 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:51.978708    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:36:51.979423 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:51.979405    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:36:52.269709 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.269620    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"]
Sep 06 10:36:52.269947 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.269896    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb" podUID="7d080f07-4766-4613-847c-009949551f10" containerName="payload-processing" containerID="cri-o://7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199" gracePeriod=30
Sep 06 10:36:52.280360 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.279245    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"]
Sep 06 10:36:52.280360 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.279748    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84" podUID="4eb2ba87-0405-45bf-9e1a-5a06472d7bc3" containerName="payload-pre-processing" containerID="cri-o://7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e" gracePeriod=30
Sep 06 10:36:52.547932 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.547906    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:36:52.550686 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.550663    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:36:52.561567 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.561537    2564 generic.go:356] "Generic (PLEG): container finished" podID="7d080f07-4766-4613-847c-009949551f10" containerID="7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199" exitCode=0
Sep 06 10:36:52.561724 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.561571    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"
Sep 06 10:36:52.561724 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.561619    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb" event={"ID":"7d080f07-4766-4613-847c-009949551f10","Type":"ContainerDied","Data":"7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199"}
Sep 06 10:36:52.561724 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.561667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb" event={"ID":"7d080f07-4766-4613-847c-009949551f10","Type":"ContainerDied","Data":"f49071e8027aaea674e55ed0931a556aa86a8e3393a4a33518c84f30670af366"}
Sep 06 10:36:52.561724 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.561692    2564 scope.go:117] "RemoveContainer" containerID="7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199"
Sep 06 10:36:52.562931 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.562908    2564 generic.go:356] "Generic (PLEG): container finished" podID="4eb2ba87-0405-45bf-9e1a-5a06472d7bc3" containerID="7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e" exitCode=0
Sep 06 10:36:52.563037 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.562941    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84" event={"ID":"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3","Type":"ContainerDied","Data":"7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e"}
Sep 06 10:36:52.563037 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.562966    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"
Sep 06 10:36:52.563037 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.562999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84" event={"ID":"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3","Type":"ContainerDied","Data":"31eec51822bb3b81f90d9c00ab5a3b71add8cf7c3f895061a67def9bddd31330"}
Sep 06 10:36:52.576011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.575450    2564 scope.go:117] "RemoveContainer" containerID="7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199"
Sep 06 10:36:52.577008 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:52.576138    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199\": container with ID starting with 7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199 not found: ID does not exist" containerID="7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199"
Sep 06 10:36:52.577008 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.576175    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199"} err="failed to get container status \"7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199\": rpc error: code = NotFound desc = could not find container \"7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199\": container with ID starting with 7673d1389ba9676edbd0ca2a44e7efe6ebb98250a0ad0032db92354473af2199 not found: ID does not exist"
Sep 06 10:36:52.577008 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.576202    2564 scope.go:117] "RemoveContainer" containerID="7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e"
Sep 06 10:36:52.592523 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.592491    2564 scope.go:117] "RemoveContainer" containerID="7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e"
Sep 06 10:36:52.592978 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:52.592952    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e\": container with ID starting with 7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e not found: ID does not exist" containerID="7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e"
Sep 06 10:36:52.593091 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.592982    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e"} err="failed to get container status \"7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e\": rpc error: code = NotFound desc = could not find container \"7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e\": container with ID starting with 7843caf04838fc05f7484969701101bd76bf9e933ea94831059d01ae5e151b3e not found: ID does not exist"
Sep 06 10:36:52.617386 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.617349    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d080f07-4766-4613-847c-009949551f10-plugins-config-volume\") pod \"7d080f07-4766-4613-847c-009949551f10\" (UID: \"7d080f07-4766-4613-847c-009949551f10\") "
Sep 06 10:36:52.617386 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.617391    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkmcn\" (UniqueName: \"kubernetes.io/projected/7d080f07-4766-4613-847c-009949551f10-kube-api-access-qkmcn\") pod \"7d080f07-4766-4613-847c-009949551f10\" (UID: \"7d080f07-4766-4613-847c-009949551f10\") "
Sep 06 10:36:52.617616 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.617432    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-plugins-config-volume\") pod \"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3\" (UID: \"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3\") "
Sep 06 10:36:52.617616 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.617456    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbfq6\" (UniqueName: \"kubernetes.io/projected/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-kube-api-access-xbfq6\") pod \"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3\" (UID: \"4eb2ba87-0405-45bf-9e1a-5a06472d7bc3\") "
Sep 06 10:36:52.617757 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.617732    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d080f07-4766-4613-847c-009949551f10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d080f07-4766-4613-847c-009949551f10" (UID: "7d080f07-4766-4613-847c-009949551f10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:36:52.617824 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.617797    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4eb2ba87-0405-45bf-9e1a-5a06472d7bc3" (UID: "4eb2ba87-0405-45bf-9e1a-5a06472d7bc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:36:52.617911 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.617896    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d080f07-4766-4613-847c-009949551f10-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:36:52.617955 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.617913    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:36:52.619687 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.619663    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d080f07-4766-4613-847c-009949551f10-kube-api-access-qkmcn" (OuterVolumeSpecName: "kube-api-access-qkmcn") pod "7d080f07-4766-4613-847c-009949551f10" (UID: "7d080f07-4766-4613-847c-009949551f10"). InnerVolumeSpecName "kube-api-access-qkmcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:36:52.619779 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.619695    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-kube-api-access-xbfq6" (OuterVolumeSpecName: "kube-api-access-xbfq6") pod "4eb2ba87-0405-45bf-9e1a-5a06472d7bc3" (UID: "4eb2ba87-0405-45bf-9e1a-5a06472d7bc3"). InnerVolumeSpecName "kube-api-access-xbfq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:36:52.718959 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.718912    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkmcn\" (UniqueName: \"kubernetes.io/projected/7d080f07-4766-4613-847c-009949551f10-kube-api-access-qkmcn\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:36:52.718959 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.718945    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbfq6\" (UniqueName: \"kubernetes.io/projected/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3-kube-api-access-xbfq6\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:36:52.888299 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.888267    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"]
Sep 06 10:36:52.892286 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.892255    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6c8f35-74646d9d4b-jvd84"]
Sep 06 10:36:52.902603 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.902572    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"]
Sep 06 10:36:52.906788 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:52.906761    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6c8f35-c8cc9cf4f-q9rkb"]
Sep 06 10:36:54.028806 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:54.028769    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb2ba87-0405-45bf-9e1a-5a06472d7bc3" path="/var/lib/kubelet/pods/4eb2ba87-0405-45bf-9e1a-5a06472d7bc3/volumes"
Sep 06 10:36:54.029198 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:54.029172    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d080f07-4766-4613-847c-009949551f10" path="/var/lib/kubelet/pods/7d080f07-4766-4613-847c-009949551f10/volumes"
Sep 06 10:36:57.266803 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:57.266766    2564 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-self-signed-certs: secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:57.267233 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:57.266835    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs podName:fcf2da22-b99a-4ea8-b584-4996795c062c nodeName:}" failed. No retries permitted until 2026-09-06 10:37:13.266821707 +0000 UTC m=+921.750321795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs") pod "explicit-model-6c8f35-kserve-599c5cf499-4c62d" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c") : secret "explicit-model-6c8f35-kserve-self-signed-certs" not found
Sep 06 10:36:57.525100 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.524974    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d" podUID="fcf2da22-b99a-4ea8-b584-4996795c062c" containerName="main" containerID="cri-o://c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76" gracePeriod=45
Sep 06 10:36:57.779155 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.779090    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:57.872044 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872002    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs\") pod \"fcf2da22-b99a-4ea8-b584-4996795c062c\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") "
Sep 06 10:36:57.872283 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872071    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-dshm\") pod \"fcf2da22-b99a-4ea8-b584-4996795c062c\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") "
Sep 06 10:36:57.872283 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872143    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-tmp-dir\") pod \"fcf2da22-b99a-4ea8-b584-4996795c062c\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") "
Sep 06 10:36:57.872283 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872176    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-model-cache\") pod \"fcf2da22-b99a-4ea8-b584-4996795c062c\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") "
Sep 06 10:36:57.872283 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872206    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-home\") pod \"fcf2da22-b99a-4ea8-b584-4996795c062c\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") "
Sep 06 10:36:57.872503 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872293    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvt9r\" (UniqueName: \"kubernetes.io/projected/fcf2da22-b99a-4ea8-b584-4996795c062c-kube-api-access-hvt9r\") pod \"fcf2da22-b99a-4ea8-b584-4996795c062c\" (UID: \"fcf2da22-b99a-4ea8-b584-4996795c062c\") "
Sep 06 10:36:57.872503 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872408    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcf2da22-b99a-4ea8-b584-4996795c062c" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:36:57.872503 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872486    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcf2da22-b99a-4ea8-b584-4996795c062c" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:36:57.872661 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872600    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-home" (OuterVolumeSpecName: "home") pod "fcf2da22-b99a-4ea8-b584-4996795c062c" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:36:57.872721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872704    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:36:57.872770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872726    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:36:57.872770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.872739    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:36:57.874422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.874388    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf2da22-b99a-4ea8-b584-4996795c062c-kube-api-access-hvt9r" (OuterVolumeSpecName: "kube-api-access-hvt9r") pod "fcf2da22-b99a-4ea8-b584-4996795c062c" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c"). InnerVolumeSpecName "kube-api-access-hvt9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:36:57.874422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.874402    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcf2da22-b99a-4ea8-b584-4996795c062c" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:36:57.874581 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.874435    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-dshm" (OuterVolumeSpecName: "dshm") pod "fcf2da22-b99a-4ea8-b584-4996795c062c" (UID: "fcf2da22-b99a-4ea8-b584-4996795c062c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:36:57.973887 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.973853    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvt9r\" (UniqueName: \"kubernetes.io/projected/fcf2da22-b99a-4ea8-b584-4996795c062c-kube-api-access-hvt9r\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:36:57.973887 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.973884    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf2da22-b99a-4ea8-b584-4996795c062c-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:36:57.973887 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:57.973893    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf2da22-b99a-4ea8-b584-4996795c062c-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:36:58.595924 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:58.595888    2564 generic.go:356] "Generic (PLEG): container finished" podID="fcf2da22-b99a-4ea8-b584-4996795c062c" containerID="c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76" exitCode=0
Sep 06 10:36:58.596389 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:58.595932    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"
Sep 06 10:36:58.596389 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:58.595941    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d" event={"ID":"fcf2da22-b99a-4ea8-b584-4996795c062c","Type":"ContainerDied","Data":"c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76"}
Sep 06 10:36:58.596389 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:58.595979    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d" event={"ID":"fcf2da22-b99a-4ea8-b584-4996795c062c","Type":"ContainerDied","Data":"bb95b90eab98fbb50e3727b994a1ffaa0fa1a79fef26a4c29e9e9dfcbe561267"}
Sep 06 10:36:58.596389 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:58.596002    2564 scope.go:117] "RemoveContainer" containerID="c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76"
Sep 06 10:36:58.604557 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:58.604535    2564 scope.go:117] "RemoveContainer" containerID="c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76"
Sep 06 10:36:58.604846 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:36:58.604825    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76\": container with ID starting with c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76 not found: ID does not exist" containerID="c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76"
Sep 06 10:36:58.604932 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:58.604858    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76"} err="failed to get container status \"c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76\": rpc error: code = NotFound desc = could not find container \"c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76\": container with ID starting with c02bb7c588d2d15284f3202dee164c6b7d3f95825f07f7a0e10e355511d3df76 not found: ID does not exist"
Sep 06 10:36:58.613871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:58.613846    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"]
Sep 06 10:36:58.615639 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:36:58.615620    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-6c8f35/explicit-model-6c8f35-kserve-599c5cf499-4c62d"]
Sep 06 10:37:00.029095 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:00.029060    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf2da22-b99a-4ea8-b584-4996795c062c" path="/var/lib/kubelet/pods/fcf2da22-b99a-4ea8-b584-4996795c062c/volumes"
Sep 06 10:37:03.954022 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:03.953984    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"]
Sep 06 10:37:03.954552 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:03.954293    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy" containerID="cri-o://15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94" gracePeriod=30
Sep 06 10:37:04.714446 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.714412    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"]
Sep 06 10:37:04.714824 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.714810    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf2da22-b99a-4ea8-b584-4996795c062c" containerName="main"
Sep 06 10:37:04.714871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.714828    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf2da22-b99a-4ea8-b584-4996795c062c" containerName="main"
Sep 06 10:37:04.714871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.714846    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d080f07-4766-4613-847c-009949551f10" containerName="payload-processing"
Sep 06 10:37:04.714871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.714852    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d080f07-4766-4613-847c-009949551f10" containerName="payload-processing"
Sep 06 10:37:04.714871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.714868    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb2ba87-0405-45bf-9e1a-5a06472d7bc3" containerName="payload-pre-processing"
Sep 06 10:37:04.715006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.714874    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb2ba87-0405-45bf-9e1a-5a06472d7bc3" containerName="payload-pre-processing"
Sep 06 10:37:04.715006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.714938    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d080f07-4766-4613-847c-009949551f10" containerName="payload-processing"
Sep 06 10:37:04.715006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.714958    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb2ba87-0405-45bf-9e1a-5a06472d7bc3" containerName="payload-pre-processing"
Sep 06 10:37:04.715006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.714968    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf2da22-b99a-4ea8-b584-4996795c062c" containerName="main"
Sep 06 10:37:04.719707 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.719685    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.722428 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.722405    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-d8ac47-openshift-default-dockercfg-f9sk5\""
Sep 06 10:37:04.729387 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.729360    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"]
Sep 06 10:37:04.739912 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.739801    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-istio-token\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.739912 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.739860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96f1686-0bd2-443f-8932-ca838a144a31-istiod-ca-cert\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.739912 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.739891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-credential-socket\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.740103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.739941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8swd\" (UniqueName: \"kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-kube-api-access-b8swd\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.740103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.739980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-envoy\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.740103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.740004    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-data\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.740103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.740038    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-certs\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.740319 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.740142    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96f1686-0bd2-443f-8932-ca838a144a31-istio-podinfo\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.740319 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.740178    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-socket\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.841480 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.841442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96f1686-0bd2-443f-8932-ca838a144a31-istio-podinfo\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.841692 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.841495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-socket\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.841692 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.841537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-istio-token\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.841692 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.841569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96f1686-0bd2-443f-8932-ca838a144a31-istiod-ca-cert\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.841692 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.841600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-credential-socket\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.841692 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.841646    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8swd\" (UniqueName: \"kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-kube-api-access-b8swd\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.841692 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.841678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-envoy\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.842031 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.841702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-data\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.842031 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.841735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-certs\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.842144 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.842024    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-socket\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.842330 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.842291    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-credential-socket\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.842436 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.842380    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-certs\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.842540 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.842515    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-data\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.842669 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.842649    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96f1686-0bd2-443f-8932-ca838a144a31-istiod-ca-cert\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.844033 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.844013    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96f1686-0bd2-443f-8932-ca838a144a31-istio-podinfo\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.844233 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.844203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-envoy\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.849791 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.849770    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8swd\" (UniqueName: \"kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-kube-api-access-b8swd\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:04.849897 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:04.849787    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-istio-token\") pod \"e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") " pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:05.031597 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:05.031505    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:05.311559 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:05.311535    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"]
Sep 06 10:37:05.313661 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:37:05.313637    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96f1686_0bd2_443f_8932_ca838a144a31.slice/crio-eef2310f9829a8aa22b33775be4c8aaab936a2cdc6ace9664cc9b2a98b86c180 WatchSource:0}: Error finding container eef2310f9829a8aa22b33775be4c8aaab936a2cdc6ace9664cc9b2a98b86c180: Status 404 returned error can't find the container with id eef2310f9829a8aa22b33775be4c8aaab936a2cdc6ace9664cc9b2a98b86c180
Sep 06 10:37:05.316013 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:05.315980    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:37:05.316097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:05.316044    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:37:05.316097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:05.316075    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:37:05.635549 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:05.635457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r" event={"ID":"b96f1686-0bd2-443f-8932-ca838a144a31","Type":"ContainerStarted","Data":"2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c"}
Sep 06 10:37:05.635549 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:05.635514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r" event={"ID":"b96f1686-0bd2-443f-8932-ca838a144a31","Type":"ContainerStarted","Data":"eef2310f9829a8aa22b33775be4c8aaab936a2cdc6ace9664cc9b2a98b86c180"}
Sep 06 10:37:05.654420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:05.654370    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r" podStartSLOduration=1.654352261 podStartE2EDuration="1.654352261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:37:05.653328431 +0000 UTC m=+914.136828564" watchObservedRunningTime="2026-09-06 10:37:05.654352261 +0000 UTC m=+914.137852369"
Sep 06 10:37:06.032425 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:06.032345    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:06.034072 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:06.034045    2564 patch_prober.go:28] interesting pod/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 06 10:37:06.034202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:06.034101    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r" podUID="b96f1686-0bd2-443f-8932-ca838a144a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 06 10:37:07.036890 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:07.036856    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:07.646185 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:07.646158    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:07.647096 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:07.647076    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:08.314839 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:08.314803    2564 patch_prober.go:28] interesting pod/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 06 10:37:08.315243 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:08.314862    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 10:37:09.204540 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.204517    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:37:09.282976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.282902    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-data\") pod \"ac056971-b231-4655-9543-b5ab8aca8311\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") "
Sep 06 10:37:09.282976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.282955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-envoy\") pod \"ac056971-b231-4655-9543-b5ab8aca8311\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") "
Sep 06 10:37:09.283192 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.282978    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-socket\") pod \"ac056971-b231-4655-9543-b5ab8aca8311\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") "
Sep 06 10:37:09.283192 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.282995    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-istio-token\") pod \"ac056971-b231-4655-9543-b5ab8aca8311\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") "
Sep 06 10:37:09.283192 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283041    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac056971-b231-4655-9543-b5ab8aca8311-istio-podinfo\") pod \"ac056971-b231-4655-9543-b5ab8aca8311\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") "
Sep 06 10:37:09.283192 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283069    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxgzl\" (UniqueName: \"kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-kube-api-access-zxgzl\") pod \"ac056971-b231-4655-9543-b5ab8aca8311\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") "
Sep 06 10:37:09.283192 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283099    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac056971-b231-4655-9543-b5ab8aca8311-istiod-ca-cert\") pod \"ac056971-b231-4655-9543-b5ab8aca8311\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") "
Sep 06 10:37:09.283192 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283172    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-credential-socket\") pod \"ac056971-b231-4655-9543-b5ab8aca8311\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") "
Sep 06 10:37:09.283544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283187    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac056971-b231-4655-9543-b5ab8aca8311" (UID: "ac056971-b231-4655-9543-b5ab8aca8311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:09.283544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283204    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-certs\") pod \"ac056971-b231-4655-9543-b5ab8aca8311\" (UID: \"ac056971-b231-4655-9543-b5ab8aca8311\") "
Sep 06 10:37:09.283544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283363    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac056971-b231-4655-9543-b5ab8aca8311" (UID: "ac056971-b231-4655-9543-b5ab8aca8311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:09.283544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283451    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac056971-b231-4655-9543-b5ab8aca8311" (UID: "ac056971-b231-4655-9543-b5ab8aca8311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:09.283758 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283585    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac056971-b231-4655-9543-b5ab8aca8311" (UID: "ac056971-b231-4655-9543-b5ab8aca8311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:09.283758 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283656    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac056971-b231-4655-9543-b5ab8aca8311-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac056971-b231-4655-9543-b5ab8aca8311" (UID: "ac056971-b231-4655-9543-b5ab8aca8311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:37:09.283758 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283691    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-credential-socket\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:09.283758 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283708    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:09.283758 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283716    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-data\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:09.283758 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.283727    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-workload-socket\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:09.285440 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.285408    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-kube-api-access-zxgzl" (OuterVolumeSpecName: "kube-api-access-zxgzl") pod "ac056971-b231-4655-9543-b5ab8aca8311" (UID: "ac056971-b231-4655-9543-b5ab8aca8311"). InnerVolumeSpecName "kube-api-access-zxgzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:09.285440 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.285424    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac056971-b231-4655-9543-b5ab8aca8311-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac056971-b231-4655-9543-b5ab8aca8311" (UID: "ac056971-b231-4655-9543-b5ab8aca8311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 10:37:09.285613 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.285570    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac056971-b231-4655-9543-b5ab8aca8311" (UID: "ac056971-b231-4655-9543-b5ab8aca8311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:09.285652 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.285628    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac056971-b231-4655-9543-b5ab8aca8311" (UID: "ac056971-b231-4655-9543-b5ab8aca8311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:09.384519 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.384490    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac056971-b231-4655-9543-b5ab8aca8311-istio-envoy\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:09.384519 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.384514    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-istio-token\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:09.384519 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.384523    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac056971-b231-4655-9543-b5ab8aca8311-istio-podinfo\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:09.384923 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.384531    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxgzl\" (UniqueName: \"kubernetes.io/projected/ac056971-b231-4655-9543-b5ab8aca8311-kube-api-access-zxgzl\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:09.384923 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.384542    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac056971-b231-4655-9543-b5ab8aca8311-istiod-ca-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:09.657445 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.657409    2564 generic.go:356] "Generic (PLEG): container finished" podID="ac056971-b231-4655-9543-b5ab8aca8311" containerID="15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94" exitCode=0
Sep 06 10:37:09.657585 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.657454    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"
Sep 06 10:37:09.657585 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.657493    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" event={"ID":"ac056971-b231-4655-9543-b5ab8aca8311","Type":"ContainerDied","Data":"15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94"}
Sep 06 10:37:09.657585 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.657542    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5" event={"ID":"ac056971-b231-4655-9543-b5ab8aca8311","Type":"ContainerDied","Data":"246becc337596fede60ba345de69a18b8a026a5a1abd382335865e9eabcf407c"}
Sep 06 10:37:09.657585 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.657565    2564 scope.go:117] "RemoveContainer" containerID="15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94"
Sep 06 10:37:09.667392 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.667322    2564 scope.go:117] "RemoveContainer" containerID="800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31"
Sep 06 10:37:09.675444 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.675424    2564 scope.go:117] "RemoveContainer" containerID="15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94"
Sep 06 10:37:09.675734 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:37:09.675716    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94\": container with ID starting with 15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94 not found: ID does not exist" containerID="15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94"
Sep 06 10:37:09.675785 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.675744    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94"} err="failed to get container status \"15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94\": rpc error: code = NotFound desc = could not find container \"15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94\": container with ID starting with 15e749e23ba172cf30a19670f58c243133eb00ab68a4cdf194d12fbf8f015a94 not found: ID does not exist"
Sep 06 10:37:09.675785 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.675760    2564 scope.go:117] "RemoveContainer" containerID="800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31"
Sep 06 10:37:09.676008 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:37:09.675987    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31\": container with ID starting with 800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31 not found: ID does not exist" containerID="800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31"
Sep 06 10:37:09.676068 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.676017    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31"} err="failed to get container status \"800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31\": rpc error: code = NotFound desc = could not find container \"800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31\": container with ID starting with 800dfbcf2f8993f0a8cd777abf55676410f470dab34ca20ae74be3e795471e31 not found: ID does not exist"
Sep 06 10:37:09.679319 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.679297    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"]
Sep 06 10:37:09.683103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:09.683081    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-6c8f35-openshift-default-6d988977c6-jvrd5"]
Sep 06 10:37:10.028410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:10.028318    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac056971-b231-4655-9543-b5ab8aca8311" path="/var/lib/kubelet/pods/ac056971-b231-4655-9543-b5ab8aca8311/volumes"
Sep 06 10:37:13.018400 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.018368    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"]
Sep 06 10:37:13.018814 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.018756    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy"
Sep 06 10:37:13.018814 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.018770    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy"
Sep 06 10:37:13.018814 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.018784    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy"
Sep 06 10:37:13.018814 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.018789    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy"
Sep 06 10:37:13.018963 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.018859    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy"
Sep 06 10:37:13.018963 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.018866    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac056971-b231-4655-9543-b5ab8aca8311" containerName="istio-proxy"
Sep 06 10:37:13.023332 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.023312    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:13.025882 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.025857    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f3881d63\""
Sep 06 10:37:13.025972 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.025919    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f3881d63-dockercfg-j9xbg\""
Sep 06 10:37:13.032333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.032311    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"]
Sep 06 10:37:13.055391 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.055360    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"]
Sep 06 10:37:13.058784 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.058767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:13.070098 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.070072    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"]
Sep 06 10:37:13.120964 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.120933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzx7l\" (UniqueName: \"kubernetes.io/projected/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-kube-api-access-hzx7l\") pod \"payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x\" (UID: \"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:13.121145 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.120978    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64723213-4609-4773-b9b2-ffc870fd8141-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5\" (UID: \"64723213-4609-4773-b9b2-ffc870fd8141\") " pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:13.121145 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.121028    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkp4w\" (UniqueName: \"kubernetes.io/projected/64723213-4609-4773-b9b2-ffc870fd8141-kube-api-access-jkp4w\") pod \"payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5\" (UID: \"64723213-4609-4773-b9b2-ffc870fd8141\") " pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:13.121145 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.121073    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x\" (UID: \"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:13.221986 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.221928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkp4w\" (UniqueName: \"kubernetes.io/projected/64723213-4609-4773-b9b2-ffc870fd8141-kube-api-access-jkp4w\") pod \"payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5\" (UID: \"64723213-4609-4773-b9b2-ffc870fd8141\") " pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:13.221986 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.221990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x\" (UID: \"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:13.222298 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.222060    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzx7l\" (UniqueName: \"kubernetes.io/projected/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-kube-api-access-hzx7l\") pod \"payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x\" (UID: \"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:13.222298 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.222084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64723213-4609-4773-b9b2-ffc870fd8141-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5\" (UID: \"64723213-4609-4773-b9b2-ffc870fd8141\") " pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:13.222770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.222746    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64723213-4609-4773-b9b2-ffc870fd8141-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5\" (UID: \"64723213-4609-4773-b9b2-ffc870fd8141\") " pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:13.222770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.222758    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x\" (UID: \"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:13.230022 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.229996    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzx7l\" (UniqueName: \"kubernetes.io/projected/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-kube-api-access-hzx7l\") pod \"payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x\" (UID: \"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:13.230154 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.230132    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkp4w\" (UniqueName: \"kubernetes.io/projected/64723213-4609-4773-b9b2-ffc870fd8141-kube-api-access-jkp4w\") pod \"payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5\" (UID: \"64723213-4609-4773-b9b2-ffc870fd8141\") " pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:13.335279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.335235    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:13.369658 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.369620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:13.610048 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.610018    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"]
Sep 06 10:37:13.611271 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:37:13.611245    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c23ce5_febe_4ee7_adaf_ae5ef0ba813d.slice/crio-6dfb0535511f1aa1b5a71cffccd3ec62ea70f3bd3e8b178c8d9f3a1087eaec7e WatchSource:0}: Error finding container 6dfb0535511f1aa1b5a71cffccd3ec62ea70f3bd3e8b178c8d9f3a1087eaec7e: Status 404 returned error can't find the container with id 6dfb0535511f1aa1b5a71cffccd3ec62ea70f3bd3e8b178c8d9f3a1087eaec7e
Sep 06 10:37:13.651111 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.651085    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"]
Sep 06 10:37:13.652966 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:37:13.652924    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64723213_4609_4773_b9b2_ffc870fd8141.slice/crio-17ba6607749f4c70591c057ecff1b1827f576419bcd153829c950405a94f041b WatchSource:0}: Error finding container 17ba6607749f4c70591c057ecff1b1827f576419bcd153829c950405a94f041b: Status 404 returned error can't find the container with id 17ba6607749f4c70591c057ecff1b1827f576419bcd153829c950405a94f041b
Sep 06 10:37:13.684578 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.684545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5" event={"ID":"64723213-4609-4773-b9b2-ffc870fd8141","Type":"ContainerStarted","Data":"17ba6607749f4c70591c057ecff1b1827f576419bcd153829c950405a94f041b"}
Sep 06 10:37:13.686184 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.686156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x" event={"ID":"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d","Type":"ContainerStarted","Data":"313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158"}
Sep 06 10:37:13.686299 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.686238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x" event={"ID":"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d","Type":"ContainerStarted","Data":"6dfb0535511f1aa1b5a71cffccd3ec62ea70f3bd3e8b178c8d9f3a1087eaec7e"}
Sep 06 10:37:13.686299 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.686277    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:13.703584 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:13.703540    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x" podStartSLOduration=0.703522901 podStartE2EDuration="703.522901ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:37:13.702853116 +0000 UTC m=+922.186353241" watchObservedRunningTime="2026-09-06 10:37:13.703522901 +0000 UTC m=+922.187023059"
Sep 06 10:37:14.692669 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:14.692634    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5" event={"ID":"64723213-4609-4773-b9b2-ffc870fd8141","Type":"ContainerStarted","Data":"5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4"}
Sep 06 10:37:14.710168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:14.710118    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5" podStartSLOduration=1.710102328 podStartE2EDuration="1.710102328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:37:14.708391378 +0000 UTC m=+923.191891505" watchObservedRunningTime="2026-09-06 10:37:14.710102328 +0000 UTC m=+923.193602440"
Sep 06 10:37:15.697434 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:15.697403    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:16.779760 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.779719    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"]
Sep 06 10:37:16.783355 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.783332    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.787700 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.787388    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d8ac47\"/\"default-dockercfg-5m5q6\""
Sep 06 10:37:16.787700 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.787661    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d8ac47\"/\"noten-model-d8ac47-kserve-self-signed-certs\""
Sep 06 10:37:16.787885 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.787841    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d8ac47\"/\"openshift-service-ca.crt\""
Sep 06 10:37:16.787931 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.787885    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d8ac47\"/\"kube-root-ca.crt\""
Sep 06 10:37:16.794791 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.794769    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"]
Sep 06 10:37:16.858838 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.858805    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhksr\" (UniqueName: \"kubernetes.io/projected/44bd273d-3857-4632-9369-ccb0802c6b5f-kube-api-access-dhksr\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.858987 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.858852    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-home\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.858987 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.858880    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-tmp-dir\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.859090 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.859005    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-model-cache\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.859090 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.859042    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-dshm\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.859153 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.859131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bd273d-3857-4632-9369-ccb0802c6b5f-tls-certs\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.960629 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.960594    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bd273d-3857-4632-9369-ccb0802c6b5f-tls-certs\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.960775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.960653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhksr\" (UniqueName: \"kubernetes.io/projected/44bd273d-3857-4632-9369-ccb0802c6b5f-kube-api-access-dhksr\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.960886 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.960824    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-home\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.960970 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.960921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-tmp-dir\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.961030 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.961008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-model-cache\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.961096 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.961038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-dshm\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.961277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.961253    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-home\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.961392 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.961337    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-tmp-dir\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.961526 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.961500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-model-cache\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.963292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.963272    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-dshm\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.963456 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.963436    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bd273d-3857-4632-9369-ccb0802c6b5f-tls-certs\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:16.968637 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:16.968607    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhksr\" (UniqueName: \"kubernetes.io/projected/44bd273d-3857-4632-9369-ccb0802c6b5f-kube-api-access-dhksr\") pod \"noten-model-d8ac47-kserve-84656d6bf9-tvf2l\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") " pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:17.100231 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:17.100164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:17.370892 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:17.370869    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"]
Sep 06 10:37:17.373354 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:37:17.373320    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44bd273d_3857_4632_9369_ccb0802c6b5f.slice/crio-a540a6b34c80d5e06c01e9de07d136f3338a1877d6ff47ba460b33ef6f31d9bd WatchSource:0}: Error finding container a540a6b34c80d5e06c01e9de07d136f3338a1877d6ff47ba460b33ef6f31d9bd: Status 404 returned error can't find the container with id a540a6b34c80d5e06c01e9de07d136f3338a1877d6ff47ba460b33ef6f31d9bd
Sep 06 10:37:17.707913 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:17.707825    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l" event={"ID":"44bd273d-3857-4632-9369-ccb0802c6b5f","Type":"ContainerStarted","Data":"a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428"}
Sep 06 10:37:17.707913 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:17.707868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l" event={"ID":"44bd273d-3857-4632-9369-ccb0802c6b5f","Type":"ContainerStarted","Data":"a540a6b34c80d5e06c01e9de07d136f3338a1877d6ff47ba460b33ef6f31d9bd"}
Sep 06 10:37:17.724414 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:17.724367    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l" podStartSLOduration=1.724351865 podStartE2EDuration="1.724351865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:37:17.723092151 +0000 UTC m=+926.206592262" watchObservedRunningTime="2026-09-06 10:37:17.724351865 +0000 UTC m=+926.207851973"
Sep 06 10:37:24.695413 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:24.695378    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:26.704118 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:26.704088    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:27.100792 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:27.100754    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:27.100972 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:27.100824    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:27.113507 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:27.113470    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:27.772274 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:27.772242    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:37:43.785596 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:43.785563    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"]
Sep 06 10:37:44.901120 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:44.901086    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"]
Sep 06 10:37:45.015823 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:45.015789    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"]
Sep 06 10:37:45.016062 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:45.016030    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r" podUID="b96f1686-0bd2-443f-8932-ca838a144a31" containerName="istio-proxy" containerID="cri-o://2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c" gracePeriod=30
Sep 06 10:37:45.976106 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:45.976075    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"]
Sep 06 10:37:50.268136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.268111    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:50.393292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393253    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-socket\") pod \"b96f1686-0bd2-443f-8932-ca838a144a31\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") "
Sep 06 10:37:50.393292 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393296    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-credential-socket\") pod \"b96f1686-0bd2-443f-8932-ca838a144a31\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") "
Sep 06 10:37:50.393574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393316    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-data\") pod \"b96f1686-0bd2-443f-8932-ca838a144a31\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") "
Sep 06 10:37:50.393574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393377    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8swd\" (UniqueName: \"kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-kube-api-access-b8swd\") pod \"b96f1686-0bd2-443f-8932-ca838a144a31\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") "
Sep 06 10:37:50.393574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393454    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-istio-token\") pod \"b96f1686-0bd2-443f-8932-ca838a144a31\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") "
Sep 06 10:37:50.393574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393492    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96f1686-0bd2-443f-8932-ca838a144a31-istio-podinfo\") pod \"b96f1686-0bd2-443f-8932-ca838a144a31\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") "
Sep 06 10:37:50.393574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393528    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96f1686-0bd2-443f-8932-ca838a144a31-istiod-ca-cert\") pod \"b96f1686-0bd2-443f-8932-ca838a144a31\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") "
Sep 06 10:37:50.393830 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393617    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b96f1686-0bd2-443f-8932-ca838a144a31" (UID: "b96f1686-0bd2-443f-8932-ca838a144a31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:50.393830 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393636    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-envoy\") pod \"b96f1686-0bd2-443f-8932-ca838a144a31\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") "
Sep 06 10:37:50.393830 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393651    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b96f1686-0bd2-443f-8932-ca838a144a31" (UID: "b96f1686-0bd2-443f-8932-ca838a144a31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:50.393830 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393693    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-certs\") pod \"b96f1686-0bd2-443f-8932-ca838a144a31\" (UID: \"b96f1686-0bd2-443f-8932-ca838a144a31\") "
Sep 06 10:37:50.394027 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393874    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-data" (OuterVolumeSpecName: "istio-data") pod "b96f1686-0bd2-443f-8932-ca838a144a31" (UID: "b96f1686-0bd2-443f-8932-ca838a144a31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:50.394027 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.393894    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96f1686-0bd2-443f-8932-ca838a144a31-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b96f1686-0bd2-443f-8932-ca838a144a31" (UID: "b96f1686-0bd2-443f-8932-ca838a144a31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:37:50.394136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.394067    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96f1686-0bd2-443f-8932-ca838a144a31-istiod-ca-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:50.394136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.394085    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-socket\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:50.394136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.394097    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-credential-socket\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:50.394136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.394113    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-data\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:50.394136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.394092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b96f1686-0bd2-443f-8932-ca838a144a31" (UID: "b96f1686-0bd2-443f-8932-ca838a144a31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:50.395873 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.395841    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-istio-token" (OuterVolumeSpecName: "istio-token") pod "b96f1686-0bd2-443f-8932-ca838a144a31" (UID: "b96f1686-0bd2-443f-8932-ca838a144a31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:50.395989 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.395883    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-kube-api-access-b8swd" (OuterVolumeSpecName: "kube-api-access-b8swd") pod "b96f1686-0bd2-443f-8932-ca838a144a31" (UID: "b96f1686-0bd2-443f-8932-ca838a144a31"). InnerVolumeSpecName "kube-api-access-b8swd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:50.395989 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.395915    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b96f1686-0bd2-443f-8932-ca838a144a31" (UID: "b96f1686-0bd2-443f-8932-ca838a144a31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:50.395989 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.395950    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b96f1686-0bd2-443f-8932-ca838a144a31-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b96f1686-0bd2-443f-8932-ca838a144a31" (UID: "b96f1686-0bd2-443f-8932-ca838a144a31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 10:37:50.495556 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.495517    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8swd\" (UniqueName: \"kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-kube-api-access-b8swd\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:50.495556 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.495548    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96f1686-0bd2-443f-8932-ca838a144a31-istio-token\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:50.495556 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.495558    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96f1686-0bd2-443f-8932-ca838a144a31-istio-podinfo\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:50.495556 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.495566    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-istio-envoy\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:50.495825 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.495576    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96f1686-0bd2-443f-8932-ca838a144a31-workload-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:50.879476 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.879441    2564 generic.go:356] "Generic (PLEG): container finished" podID="b96f1686-0bd2-443f-8932-ca838a144a31" containerID="2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c" exitCode=0
Sep 06 10:37:50.879654 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.879490    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"
Sep 06 10:37:50.879654 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.879530    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r" event={"ID":"b96f1686-0bd2-443f-8932-ca838a144a31","Type":"ContainerDied","Data":"2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c"}
Sep 06 10:37:50.879654 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.879583    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r" event={"ID":"b96f1686-0bd2-443f-8932-ca838a144a31","Type":"ContainerDied","Data":"eef2310f9829a8aa22b33775be4c8aaab936a2cdc6ace9664cc9b2a98b86c180"}
Sep 06 10:37:50.879654 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.879606    2564 scope.go:117] "RemoveContainer" containerID="2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c"
Sep 06 10:37:50.899456 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.899426    2564 scope.go:117] "RemoveContainer" containerID="2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c"
Sep 06 10:37:50.899810 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:37:50.899787    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c\": container with ID starting with 2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c not found: ID does not exist" containerID="2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c"
Sep 06 10:37:50.899896 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.899816    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c"} err="failed to get container status \"2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c\": rpc error: code = NotFound desc = could not find container \"2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c\": container with ID starting with 2c3b47483be6c14c09abfc8bb59e3e8b0569b9479f2309b3a0743dbfe86eb47c not found: ID does not exist"
Sep 06 10:37:50.909832 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.909809    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"]
Sep 06 10:37:50.914146 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:50.914122    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-d8ac47-openshift-default-64c9b6b75d-7g24r"]
Sep 06 10:37:52.029883 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:52.029850    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b96f1686-0bd2-443f-8932-ca838a144a31" path="/var/lib/kubelet/pods/b96f1686-0bd2-443f-8932-ca838a144a31/volumes"
Sep 06 10:37:54.292906 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.292866    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"]
Sep 06 10:37:54.293406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.293140    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5" podUID="64723213-4609-4773-b9b2-ffc870fd8141" containerName="payload-processing" containerID="cri-o://5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4" gracePeriod=30
Sep 06 10:37:54.298574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.298520    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"]
Sep 06 10:37:54.299086 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.298752    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x" podUID="43c23ce5-febe-4ee7-adaf-ae5ef0ba813d" containerName="payload-pre-processing" containerID="cri-o://313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158" gracePeriod=30
Sep 06 10:37:54.565916 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.565892    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:54.568747 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.568729    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:54.633851 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.633816    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-plugins-config-volume\") pod \"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d\" (UID: \"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d\") "
Sep 06 10:37:54.634033 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.633864    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64723213-4609-4773-b9b2-ffc870fd8141-plugins-config-volume\") pod \"64723213-4609-4773-b9b2-ffc870fd8141\" (UID: \"64723213-4609-4773-b9b2-ffc870fd8141\") "
Sep 06 10:37:54.634033 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.633927    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzx7l\" (UniqueName: \"kubernetes.io/projected/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-kube-api-access-hzx7l\") pod \"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d\" (UID: \"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d\") "
Sep 06 10:37:54.634033 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.633954    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkp4w\" (UniqueName: \"kubernetes.io/projected/64723213-4609-4773-b9b2-ffc870fd8141-kube-api-access-jkp4w\") pod \"64723213-4609-4773-b9b2-ffc870fd8141\" (UID: \"64723213-4609-4773-b9b2-ffc870fd8141\") "
Sep 06 10:37:54.634303 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.634276    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43c23ce5-febe-4ee7-adaf-ae5ef0ba813d" (UID: "43c23ce5-febe-4ee7-adaf-ae5ef0ba813d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:37:54.634365 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.634274    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64723213-4609-4773-b9b2-ffc870fd8141-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64723213-4609-4773-b9b2-ffc870fd8141" (UID: "64723213-4609-4773-b9b2-ffc870fd8141"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:37:54.636042 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.636018    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-kube-api-access-hzx7l" (OuterVolumeSpecName: "kube-api-access-hzx7l") pod "43c23ce5-febe-4ee7-adaf-ae5ef0ba813d" (UID: "43c23ce5-febe-4ee7-adaf-ae5ef0ba813d"). InnerVolumeSpecName "kube-api-access-hzx7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:54.636119 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.636082    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64723213-4609-4773-b9b2-ffc870fd8141-kube-api-access-jkp4w" (OuterVolumeSpecName: "kube-api-access-jkp4w") pod "64723213-4609-4773-b9b2-ffc870fd8141" (UID: "64723213-4609-4773-b9b2-ffc870fd8141"). InnerVolumeSpecName "kube-api-access-jkp4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:54.734974 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.734930    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:54.734974 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.734969    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64723213-4609-4773-b9b2-ffc870fd8141-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:54.734974 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.734978    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzx7l\" (UniqueName: \"kubernetes.io/projected/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d-kube-api-access-hzx7l\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:54.734974 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.734986    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkp4w\" (UniqueName: \"kubernetes.io/projected/64723213-4609-4773-b9b2-ffc870fd8141-kube-api-access-jkp4w\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:54.901784 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.901694    2564 generic.go:356] "Generic (PLEG): container finished" podID="64723213-4609-4773-b9b2-ffc870fd8141" containerID="5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4" exitCode=0
Sep 06 10:37:54.901784 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.901736    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"
Sep 06 10:37:54.901981 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.901778    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5" event={"ID":"64723213-4609-4773-b9b2-ffc870fd8141","Type":"ContainerDied","Data":"5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4"}
Sep 06 10:37:54.901981 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.901841    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5" event={"ID":"64723213-4609-4773-b9b2-ffc870fd8141","Type":"ContainerDied","Data":"17ba6607749f4c70591c057ecff1b1827f576419bcd153829c950405a94f041b"}
Sep 06 10:37:54.901981 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.901876    2564 scope.go:117] "RemoveContainer" containerID="5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4"
Sep 06 10:37:54.903066 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.903049    2564 generic.go:356] "Generic (PLEG): container finished" podID="43c23ce5-febe-4ee7-adaf-ae5ef0ba813d" containerID="313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158" exitCode=0
Sep 06 10:37:54.903148 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.903092    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"
Sep 06 10:37:54.903237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.903146    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x" event={"ID":"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d","Type":"ContainerDied","Data":"313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158"}
Sep 06 10:37:54.903237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.903198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x" event={"ID":"43c23ce5-febe-4ee7-adaf-ae5ef0ba813d","Type":"ContainerDied","Data":"6dfb0535511f1aa1b5a71cffccd3ec62ea70f3bd3e8b178c8d9f3a1087eaec7e"}
Sep 06 10:37:54.911543 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.911526    2564 scope.go:117] "RemoveContainer" containerID="5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4"
Sep 06 10:37:54.911887 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:37:54.911864    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4\": container with ID starting with 5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4 not found: ID does not exist" containerID="5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4"
Sep 06 10:37:54.911987 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.911898    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4"} err="failed to get container status \"5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4\": rpc error: code = NotFound desc = could not find container \"5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4\": container with ID starting with 5df9214647d9a0e2451cf1bd368cf65d20d1b7a8b31270d67b2045a3415307d4 not found: ID does not exist"
Sep 06 10:37:54.911987 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.911918    2564 scope.go:117] "RemoveContainer" containerID="313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158"
Sep 06 10:37:54.921446 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.921422    2564 scope.go:117] "RemoveContainer" containerID="313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158"
Sep 06 10:37:54.921714 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:37:54.921696    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158\": container with ID starting with 313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158 not found: ID does not exist" containerID="313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158"
Sep 06 10:37:54.921786 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.921729    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158"} err="failed to get container status \"313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158\": rpc error: code = NotFound desc = could not find container \"313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158\": container with ID starting with 313be587a9897c9479ba03eaecf32c3589e5d6de4ce4c8d59bad56071fa91158 not found: ID does not exist"
Sep 06 10:37:54.925672 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.925636    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"]
Sep 06 10:37:54.930744 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.930717    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3881d63-78b95cff97-5jkj5"]
Sep 06 10:37:54.940398 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.940376    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"]
Sep 06 10:37:54.943765 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:54.943743    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3881d63-788879f56b-8xk2x"]
Sep 06 10:37:56.028466 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:56.028434    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c23ce5-febe-4ee7-adaf-ae5ef0ba813d" path="/var/lib/kubelet/pods/43c23ce5-febe-4ee7-adaf-ae5ef0ba813d/volumes"
Sep 06 10:37:56.028854 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:56.028827    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64723213-4609-4773-b9b2-ffc870fd8141" path="/var/lib/kubelet/pods/64723213-4609-4773-b9b2-ffc870fd8141/volumes"
Sep 06 10:37:58.807087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:58.807043    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj" podUID="c9a002a7-b923-4784-a4eb-08e981f011fd" containerName="main" containerID="cri-o://d44ec3edf1222749d100fa06c31b5e5dace6cb09c48811c7968bcf1b19b87393" gracePeriod=45
Sep 06 10:37:58.928794 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:58.928760    2564 generic.go:356] "Generic (PLEG): container finished" podID="c9a002a7-b923-4784-a4eb-08e981f011fd" containerID="d44ec3edf1222749d100fa06c31b5e5dace6cb09c48811c7968bcf1b19b87393" exitCode=0
Sep 06 10:37:58.928955 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:58.928794    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj" event={"ID":"c9a002a7-b923-4784-a4eb-08e981f011fd","Type":"ContainerDied","Data":"d44ec3edf1222749d100fa06c31b5e5dace6cb09c48811c7968bcf1b19b87393"}
Sep 06 10:37:59.062992 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.062918    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:37:59.180133 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180098    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzbht\" (UniqueName: \"kubernetes.io/projected/c9a002a7-b923-4784-a4eb-08e981f011fd-kube-api-access-tzbht\") pod \"c9a002a7-b923-4784-a4eb-08e981f011fd\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") "
Sep 06 10:37:59.180321 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180145    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-dshm\") pod \"c9a002a7-b923-4784-a4eb-08e981f011fd\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") "
Sep 06 10:37:59.180321 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180229    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-tmp-dir\") pod \"c9a002a7-b923-4784-a4eb-08e981f011fd\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") "
Sep 06 10:37:59.180321 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180246    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-home\") pod \"c9a002a7-b923-4784-a4eb-08e981f011fd\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") "
Sep 06 10:37:59.180321 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180286    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-model-cache\") pod \"c9a002a7-b923-4784-a4eb-08e981f011fd\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") "
Sep 06 10:37:59.180321 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180312    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a002a7-b923-4784-a4eb-08e981f011fd-tls-certs\") pod \"c9a002a7-b923-4784-a4eb-08e981f011fd\" (UID: \"c9a002a7-b923-4784-a4eb-08e981f011fd\") "
Sep 06 10:37:59.180598 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180531    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9a002a7-b923-4784-a4eb-08e981f011fd" (UID: "c9a002a7-b923-4784-a4eb-08e981f011fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:59.180598 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180554    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9a002a7-b923-4784-a4eb-08e981f011fd" (UID: "c9a002a7-b923-4784-a4eb-08e981f011fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:59.180598 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180516    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-home" (OuterVolumeSpecName: "home") pod "c9a002a7-b923-4784-a4eb-08e981f011fd" (UID: "c9a002a7-b923-4784-a4eb-08e981f011fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:59.180752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180683    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:59.180752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180701    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:59.180752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.180714    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:59.182519 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.182491    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a002a7-b923-4784-a4eb-08e981f011fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9a002a7-b923-4784-a4eb-08e981f011fd" (UID: "c9a002a7-b923-4784-a4eb-08e981f011fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:37:59.182629 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.182508    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-dshm" (OuterVolumeSpecName: "dshm") pod "c9a002a7-b923-4784-a4eb-08e981f011fd" (UID: "c9a002a7-b923-4784-a4eb-08e981f011fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:37:59.182629 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.182502    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a002a7-b923-4784-a4eb-08e981f011fd-kube-api-access-tzbht" (OuterVolumeSpecName: "kube-api-access-tzbht") pod "c9a002a7-b923-4784-a4eb-08e981f011fd" (UID: "c9a002a7-b923-4784-a4eb-08e981f011fd"). InnerVolumeSpecName "kube-api-access-tzbht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:59.281735 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.281694    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzbht\" (UniqueName: \"kubernetes.io/projected/c9a002a7-b923-4784-a4eb-08e981f011fd-kube-api-access-tzbht\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:59.281735 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.281726    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a002a7-b923-4784-a4eb-08e981f011fd-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:59.281735 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.281737    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a002a7-b923-4784-a4eb-08e981f011fd-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:37:59.922933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.922865    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l" podUID="44bd273d-3857-4632-9369-ccb0802c6b5f" containerName="main" containerID="cri-o://a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428" gracePeriod=45
Sep 06 10:37:59.938749 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.938719    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj" event={"ID":"c9a002a7-b923-4784-a4eb-08e981f011fd","Type":"ContainerDied","Data":"8c241224ad50edf81088a195b54a9a79180da6b10f98cf98acfd4a0c224de317"}
Sep 06 10:37:59.938888 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.938751    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"
Sep 06 10:37:59.938888 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:37:59.938765    2564 scope.go:117] "RemoveContainer" containerID="d44ec3edf1222749d100fa06c31b5e5dace6cb09c48811c7968bcf1b19b87393"
Sep 06 10:38:00.001251 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.001222    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"]
Sep 06 10:38:00.004407 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.004378    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-04526a/test-model-04526a-kserve-9ddcfb9df-dxrxj"]
Sep 06 10:38:00.029251 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.029198    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a002a7-b923-4784-a4eb-08e981f011fd" path="/var/lib/kubelet/pods/c9a002a7-b923-4784-a4eb-08e981f011fd/volumes"
Sep 06 10:38:00.176823 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.176755    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:38:00.290485 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.290452    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bd273d-3857-4632-9369-ccb0802c6b5f-tls-certs\") pod \"44bd273d-3857-4632-9369-ccb0802c6b5f\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") "
Sep 06 10:38:00.290485 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.290490    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-tmp-dir\") pod \"44bd273d-3857-4632-9369-ccb0802c6b5f\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") "
Sep 06 10:38:00.290770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.290516    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-dshm\") pod \"44bd273d-3857-4632-9369-ccb0802c6b5f\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") "
Sep 06 10:38:00.290770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.290539    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-model-cache\") pod \"44bd273d-3857-4632-9369-ccb0802c6b5f\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") "
Sep 06 10:38:00.290770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.290599    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhksr\" (UniqueName: \"kubernetes.io/projected/44bd273d-3857-4632-9369-ccb0802c6b5f-kube-api-access-dhksr\") pod \"44bd273d-3857-4632-9369-ccb0802c6b5f\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") "
Sep 06 10:38:00.290770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.290629    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-home\") pod \"44bd273d-3857-4632-9369-ccb0802c6b5f\" (UID: \"44bd273d-3857-4632-9369-ccb0802c6b5f\") "
Sep 06 10:38:00.290998 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.290787    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44bd273d-3857-4632-9369-ccb0802c6b5f" (UID: "44bd273d-3857-4632-9369-ccb0802c6b5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:38:00.290998 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.290908    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:00.290998 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.290936    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "44bd273d-3857-4632-9369-ccb0802c6b5f" (UID: "44bd273d-3857-4632-9369-ccb0802c6b5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:38:00.291164 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.290997    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-home" (OuterVolumeSpecName: "home") pod "44bd273d-3857-4632-9369-ccb0802c6b5f" (UID: "44bd273d-3857-4632-9369-ccb0802c6b5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:38:00.293120 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.293084    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-dshm" (OuterVolumeSpecName: "dshm") pod "44bd273d-3857-4632-9369-ccb0802c6b5f" (UID: "44bd273d-3857-4632-9369-ccb0802c6b5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:38:00.293120 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.293098    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bd273d-3857-4632-9369-ccb0802c6b5f-kube-api-access-dhksr" (OuterVolumeSpecName: "kube-api-access-dhksr") pod "44bd273d-3857-4632-9369-ccb0802c6b5f" (UID: "44bd273d-3857-4632-9369-ccb0802c6b5f"). InnerVolumeSpecName "kube-api-access-dhksr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:38:00.293366 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.293251    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bd273d-3857-4632-9369-ccb0802c6b5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44bd273d-3857-4632-9369-ccb0802c6b5f" (UID: "44bd273d-3857-4632-9369-ccb0802c6b5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:38:00.391406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.391362    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:00.391406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.391398    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44bd273d-3857-4632-9369-ccb0802c6b5f-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:00.391406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.391408    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:00.391406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.391416    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44bd273d-3857-4632-9369-ccb0802c6b5f-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:00.391673 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.391424    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhksr\" (UniqueName: \"kubernetes.io/projected/44bd273d-3857-4632-9369-ccb0802c6b5f-kube-api-access-dhksr\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:00.945103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.945063    2564 generic.go:356] "Generic (PLEG): container finished" podID="44bd273d-3857-4632-9369-ccb0802c6b5f" containerID="a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428" exitCode=0
Sep 06 10:38:00.945538 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.945111    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"
Sep 06 10:38:00.945538 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.945143    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l" event={"ID":"44bd273d-3857-4632-9369-ccb0802c6b5f","Type":"ContainerDied","Data":"a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428"}
Sep 06 10:38:00.945538 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.945197    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l" event={"ID":"44bd273d-3857-4632-9369-ccb0802c6b5f","Type":"ContainerDied","Data":"a540a6b34c80d5e06c01e9de07d136f3338a1877d6ff47ba460b33ef6f31d9bd"}
Sep 06 10:38:00.945538 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.945244    2564 scope.go:117] "RemoveContainer" containerID="a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428"
Sep 06 10:38:00.954522 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.954506    2564 scope.go:117] "RemoveContainer" containerID="a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428"
Sep 06 10:38:00.954820 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:38:00.954798    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428\": container with ID starting with a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428 not found: ID does not exist" containerID="a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428"
Sep 06 10:38:00.954873 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.954830    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428"} err="failed to get container status \"a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428\": rpc error: code = NotFound desc = could not find container \"a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428\": container with ID starting with a7939dbd521e0d0161f7549f7324abc1664c034d911737bf7538c7136e35b428 not found: ID does not exist"
Sep 06 10:38:00.966221 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.966178    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"]
Sep 06 10:38:00.969844 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.969823    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-d8ac47/noten-model-d8ac47-kserve-84656d6bf9-tvf2l"]
Sep 06 10:38:00.996584 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:00.996551    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm" podUID="3e95ca89-abd5-4489-a938-7d364f08a68f" containerName="main" containerID="cri-o://b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05" gracePeriod=45
Sep 06 10:38:01.248708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.248678    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:38:01.401240 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401184    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-dshm\") pod \"3e95ca89-abd5-4489-a938-7d364f08a68f\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") "
Sep 06 10:38:01.401422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401246    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-home\") pod \"3e95ca89-abd5-4489-a938-7d364f08a68f\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") "
Sep 06 10:38:01.401422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401307    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e95ca89-abd5-4489-a938-7d364f08a68f-tls-certs\") pod \"3e95ca89-abd5-4489-a938-7d364f08a68f\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") "
Sep 06 10:38:01.401422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401340    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-model-cache\") pod \"3e95ca89-abd5-4489-a938-7d364f08a68f\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") "
Sep 06 10:38:01.401422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401377    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-tmp-dir\") pod \"3e95ca89-abd5-4489-a938-7d364f08a68f\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") "
Sep 06 10:38:01.401658 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401457    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q6kb\" (UniqueName: \"kubernetes.io/projected/3e95ca89-abd5-4489-a938-7d364f08a68f-kube-api-access-9q6kb\") pod \"3e95ca89-abd5-4489-a938-7d364f08a68f\" (UID: \"3e95ca89-abd5-4489-a938-7d364f08a68f\") "
Sep 06 10:38:01.401658 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401492    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-home" (OuterVolumeSpecName: "home") pod "3e95ca89-abd5-4489-a938-7d364f08a68f" (UID: "3e95ca89-abd5-4489-a938-7d364f08a68f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:38:01.401658 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401595    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e95ca89-abd5-4489-a938-7d364f08a68f" (UID: "3e95ca89-abd5-4489-a938-7d364f08a68f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:38:01.401658 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401621    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e95ca89-abd5-4489-a938-7d364f08a68f" (UID: "3e95ca89-abd5-4489-a938-7d364f08a68f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:38:01.401851 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401800    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:01.401851 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401821    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:01.401851 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.401839    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:01.403583 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.403557    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e95ca89-abd5-4489-a938-7d364f08a68f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e95ca89-abd5-4489-a938-7d364f08a68f" (UID: "3e95ca89-abd5-4489-a938-7d364f08a68f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:38:01.403583 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.403576    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-dshm" (OuterVolumeSpecName: "dshm") pod "3e95ca89-abd5-4489-a938-7d364f08a68f" (UID: "3e95ca89-abd5-4489-a938-7d364f08a68f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:38:01.403724 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.403582    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e95ca89-abd5-4489-a938-7d364f08a68f-kube-api-access-9q6kb" (OuterVolumeSpecName: "kube-api-access-9q6kb") pod "3e95ca89-abd5-4489-a938-7d364f08a68f" (UID: "3e95ca89-abd5-4489-a938-7d364f08a68f"). InnerVolumeSpecName "kube-api-access-9q6kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:38:01.503168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.503082    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q6kb\" (UniqueName: \"kubernetes.io/projected/3e95ca89-abd5-4489-a938-7d364f08a68f-kube-api-access-9q6kb\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:01.503168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.503111    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e95ca89-abd5-4489-a938-7d364f08a68f-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:01.503168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.503124    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e95ca89-abd5-4489-a938-7d364f08a68f-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:01.952843 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.952807    2564 generic.go:356] "Generic (PLEG): container finished" podID="3e95ca89-abd5-4489-a938-7d364f08a68f" containerID="b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05" exitCode=0
Sep 06 10:38:01.953339 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.952854    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"
Sep 06 10:38:01.953339 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.952886    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm" event={"ID":"3e95ca89-abd5-4489-a938-7d364f08a68f","Type":"ContainerDied","Data":"b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05"}
Sep 06 10:38:01.953339 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.952926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm" event={"ID":"3e95ca89-abd5-4489-a938-7d364f08a68f","Type":"ContainerDied","Data":"0344087bf6cfa0eb2775eaf4b5db6b821ba5b058db0b6cb59cb267fcd5c65962"}
Sep 06 10:38:01.953339 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.952952    2564 scope.go:117] "RemoveContainer" containerID="b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05"
Sep 06 10:38:01.961854 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.961836    2564 scope.go:117] "RemoveContainer" containerID="b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05"
Sep 06 10:38:01.962136 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:38:01.962115    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05\": container with ID starting with b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05 not found: ID does not exist" containerID="b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05"
Sep 06 10:38:01.962180 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.962144    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05"} err="failed to get container status \"b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05\": rpc error: code = NotFound desc = could not find container \"b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05\": container with ID starting with b1a06a4dda9da910c4bbc982fcad51ecabf92c56d6d51bd4d8c468cf7dbcfa05 not found: ID does not exist"
Sep 06 10:38:01.974007 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.973985    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"]
Sep 06 10:38:01.983973 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:01.983952    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bd3407/test-model-bd3407-kserve-57676b4c9c-wglqm"]
Sep 06 10:38:02.028599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:02.028564    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e95ca89-abd5-4489-a938-7d364f08a68f" path="/var/lib/kubelet/pods/3e95ca89-abd5-4489-a938-7d364f08a68f/volumes"
Sep 06 10:38:02.028939 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:02.028927    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44bd273d-3857-4632-9369-ccb0802c6b5f" path="/var/lib/kubelet/pods/44bd273d-3857-4632-9369-ccb0802c6b5f/volumes"
Sep 06 10:38:14.456824 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.456729    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"]
Sep 06 10:38:14.457347 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.456972    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw" podUID="e4642f16-18cb-41c9-a3fc-569a96bf7db0" containerName="maas-api" containerID="cri-o://77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050" gracePeriod=30
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.515789    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"]
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516340    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a002a7-b923-4784-a4eb-08e981f011fd" containerName="main"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516357    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a002a7-b923-4784-a4eb-08e981f011fd" containerName="main"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516369    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c23ce5-febe-4ee7-adaf-ae5ef0ba813d" containerName="payload-pre-processing"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516377    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c23ce5-febe-4ee7-adaf-ae5ef0ba813d" containerName="payload-pre-processing"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516394    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96f1686-0bd2-443f-8932-ca838a144a31" containerName="istio-proxy"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516402    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96f1686-0bd2-443f-8932-ca838a144a31" containerName="istio-proxy"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516411    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44bd273d-3857-4632-9369-ccb0802c6b5f" containerName="main"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516420    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bd273d-3857-4632-9369-ccb0802c6b5f" containerName="main"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516433    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e95ca89-abd5-4489-a938-7d364f08a68f" containerName="main"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516441    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e95ca89-abd5-4489-a938-7d364f08a68f" containerName="main"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516460    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64723213-4609-4773-b9b2-ffc870fd8141" containerName="payload-processing"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516468    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="64723213-4609-4773-b9b2-ffc870fd8141" containerName="payload-processing"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516561    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e95ca89-abd5-4489-a938-7d364f08a68f" containerName="main"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516571    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a002a7-b923-4784-a4eb-08e981f011fd" containerName="main"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516580    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c23ce5-febe-4ee7-adaf-ae5ef0ba813d" containerName="payload-pre-processing"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516594    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="64723213-4609-4773-b9b2-ffc870fd8141" containerName="payload-processing"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516603    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="44bd273d-3857-4632-9369-ccb0802c6b5f" containerName="main"
Sep 06 10:38:14.518528 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.516611    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96f1686-0bd2-443f-8932-ca838a144a31" containerName="istio-proxy"
Sep 06 10:38:14.522775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.519778    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.523277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.523245    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-27a2b9-openshift-default-dockercfg-zts7p\""
Sep 06 10:38:14.523446 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.523423    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 06 10:38:14.536613 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.536586    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"]
Sep 06 10:38:14.626346 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.626319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxhjl\" (UniqueName: \"kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-kube-api-access-vxhjl\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.626437 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.626369    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-certs\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.626503 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.626439    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-credential-socket\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.626554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.626511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-token\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.626618 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.626599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-podinfo\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.626685 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.626628    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-socket\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.626685 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.626644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-data\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.626685 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.626676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-envoy\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.626861 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.626744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istiod-ca-cert\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.701604 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.701580    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:38:14.727631 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.727540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxhjl\" (UniqueName: \"kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-kube-api-access-vxhjl\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.727631 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.727579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-certs\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.727631 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.727609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-credential-socket\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.727993 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.727736    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-token\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.727993 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.727845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-podinfo\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.727993 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.727880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-socket\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.727993 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.727909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-data\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.727993 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.727940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-envoy\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.727993 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.727987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-certs\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.728354 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.727997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istiod-ca-cert\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.728354 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.728078    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-credential-socket\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.728354 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.728284    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-data\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.728544 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.728410    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-socket\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.728693 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.728674    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istiod-ca-cert\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.730204 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.730175    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-envoy\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.730339 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.730320    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-podinfo\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.735602 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.735581    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-token\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.735792 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.735773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxhjl\" (UniqueName: \"kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-kube-api-access-vxhjl\") pod \"e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") " pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.828725 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.828689    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-maas-api-tls\") pod \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") "
Sep 06 10:38:14.828893 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.828747    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4642f16-18cb-41c9-a3fc-569a96bf7db0-openshift-service-ca\") pod \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") "
Sep 06 10:38:14.828893 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.828790    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-metrics-certs\") pod \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") "
Sep 06 10:38:14.828986 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.828904    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb88q\" (UniqueName: \"kubernetes.io/projected/e4642f16-18cb-41c9-a3fc-569a96bf7db0-kube-api-access-mb88q\") pod \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\" (UID: \"e4642f16-18cb-41c9-a3fc-569a96bf7db0\") "
Sep 06 10:38:14.829174 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.829147    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4642f16-18cb-41c9-a3fc-569a96bf7db0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4642f16-18cb-41c9-a3fc-569a96bf7db0" (UID: "e4642f16-18cb-41c9-a3fc-569a96bf7db0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:38:14.829362 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.829345    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4642f16-18cb-41c9-a3fc-569a96bf7db0-openshift-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:14.830953 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.830919    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4642f16-18cb-41c9-a3fc-569a96bf7db0" (UID: "e4642f16-18cb-41c9-a3fc-569a96bf7db0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:38:14.831057 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.830965    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e4642f16-18cb-41c9-a3fc-569a96bf7db0" (UID: "e4642f16-18cb-41c9-a3fc-569a96bf7db0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:38:14.831359 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.831340    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4642f16-18cb-41c9-a3fc-569a96bf7db0-kube-api-access-mb88q" (OuterVolumeSpecName: "kube-api-access-mb88q") pod "e4642f16-18cb-41c9-a3fc-569a96bf7db0" (UID: "e4642f16-18cb-41c9-a3fc-569a96bf7db0"). InnerVolumeSpecName "kube-api-access-mb88q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:38:14.834497 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.834482    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:14.930750 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.930718    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-metrics-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:14.930963 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.930747    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb88q\" (UniqueName: \"kubernetes.io/projected/e4642f16-18cb-41c9-a3fc-569a96bf7db0-kube-api-access-mb88q\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:14.930963 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:14.930772    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4642f16-18cb-41c9-a3fc-569a96bf7db0-maas-api-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:38:15.020201 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.020103    2564 generic.go:356] "Generic (PLEG): container finished" podID="e4642f16-18cb-41c9-a3fc-569a96bf7db0" containerID="77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050" exitCode=0
Sep 06 10:38:15.020201 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.020149    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"
Sep 06 10:38:15.020201 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.020192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw" event={"ID":"e4642f16-18cb-41c9-a3fc-569a96bf7db0","Type":"ContainerDied","Data":"77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050"}
Sep 06 10:38:15.020442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.020256    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw" event={"ID":"e4642f16-18cb-41c9-a3fc-569a96bf7db0","Type":"ContainerDied","Data":"5dbdff79447ce51947fe1c2c9650b57aa258b75805d21ee7fd0ca0ffbb371577"}
Sep 06 10:38:15.020442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.020280    2564 scope.go:117] "RemoveContainer" containerID="77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050"
Sep 06 10:38:15.037475 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.037453    2564 scope.go:117] "RemoveContainer" containerID="77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050"
Sep 06 10:38:15.037801 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:38:15.037779    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050\": container with ID starting with 77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050 not found: ID does not exist" containerID="77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050"
Sep 06 10:38:15.037863 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.037810    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050"} err="failed to get container status \"77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050\": rpc error: code = NotFound desc = could not find container \"77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050\": container with ID starting with 77fbd9310d6295c6f86a9efa671cc498acda86457ab2b27c00b6af058c9de050 not found: ID does not exist"
Sep 06 10:38:15.048372 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.048343    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"]
Sep 06 10:38:15.052696 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.052674    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bd3407-fc5df9f86-bh2hw"]
Sep 06 10:38:15.112707 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.112678    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"]
Sep 06 10:38:15.114667 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:38:15.114644    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a86f003_7ddc_4a8e_889d_bdcd3ed11239.slice/crio-545f6567980db746cc117bc65e9895807a0bd999fdad5430b97ad44d1311e1a7 WatchSource:0}: Error finding container 545f6567980db746cc117bc65e9895807a0bd999fdad5430b97ad44d1311e1a7: Status 404 returned error can't find the container with id 545f6567980db746cc117bc65e9895807a0bd999fdad5430b97ad44d1311e1a7
Sep 06 10:38:15.116806 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.116774    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:38:15.116870 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.116837    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:38:15.116908 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:15.116868    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:38:16.032194 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:16.032163    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4642f16-18cb-41c9-a3fc-569a96bf7db0" path="/var/lib/kubelet/pods/e4642f16-18cb-41c9-a3fc-569a96bf7db0/volumes"
Sep 06 10:38:16.032580 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:16.032482    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" event={"ID":"4a86f003-7ddc-4a8e-889d-bdcd3ed11239","Type":"ContainerStarted","Data":"42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3"}
Sep 06 10:38:16.032580 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:16.032530    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" event={"ID":"4a86f003-7ddc-4a8e-889d-bdcd3ed11239","Type":"ContainerStarted","Data":"545f6567980db746cc117bc65e9895807a0bd999fdad5430b97ad44d1311e1a7"}
Sep 06 10:38:16.054519 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:16.054474    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" podStartSLOduration=2.054459109 podStartE2EDuration="2.054459109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:16.052607044 +0000 UTC m=+984.536107173" watchObservedRunningTime="2026-09-06 10:38:16.054459109 +0000 UTC m=+984.537959218"
Sep 06 10:38:16.834973 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:16.834930    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:16.840281 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:16.840255    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:17.036162 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:17.036119    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:17.037410 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:17.037385    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:19.901463 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:19.901424    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"]
Sep 06 10:38:19.901855 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:19.901797    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4642f16-18cb-41c9-a3fc-569a96bf7db0" containerName="maas-api"
Sep 06 10:38:19.901855 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:19.901808    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4642f16-18cb-41c9-a3fc-569a96bf7db0" containerName="maas-api"
Sep 06 10:38:19.901933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:19.901881    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4642f16-18cb-41c9-a3fc-569a96bf7db0" containerName="maas-api"
Sep 06 10:38:19.904732 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:19.904713    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:38:19.907465 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:19.907435    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-27a2b9\""
Sep 06 10:38:19.907606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:19.907578    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-27a2b9-dockercfg-cgwjz\""
Sep 06 10:38:19.913444 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:19.913423    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"]
Sep 06 10:38:19.976301 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:19.976263    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78180f7-be5e-48dd-90e3-fb497aa53b50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg\" (UID: \"c78180f7-be5e-48dd-90e3-fb497aa53b50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:38:19.976469 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:19.976314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqxqz\" (UniqueName: \"kubernetes.io/projected/c78180f7-be5e-48dd-90e3-fb497aa53b50-kube-api-access-pqxqz\") pod \"payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg\" (UID: \"c78180f7-be5e-48dd-90e3-fb497aa53b50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:38:20.077295 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:20.077245    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78180f7-be5e-48dd-90e3-fb497aa53b50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg\" (UID: \"c78180f7-be5e-48dd-90e3-fb497aa53b50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:38:20.077472 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:20.077308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxqz\" (UniqueName: \"kubernetes.io/projected/c78180f7-be5e-48dd-90e3-fb497aa53b50-kube-api-access-pqxqz\") pod \"payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg\" (UID: \"c78180f7-be5e-48dd-90e3-fb497aa53b50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:38:20.077929 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:20.077909    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78180f7-be5e-48dd-90e3-fb497aa53b50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg\" (UID: \"c78180f7-be5e-48dd-90e3-fb497aa53b50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:38:20.085400 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:20.085371    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxqz\" (UniqueName: \"kubernetes.io/projected/c78180f7-be5e-48dd-90e3-fb497aa53b50-kube-api-access-pqxqz\") pod \"payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg\" (UID: \"c78180f7-be5e-48dd-90e3-fb497aa53b50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:38:20.216314 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:20.216198    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:38:20.484736 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:20.484640    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"]
Sep 06 10:38:20.488963 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:38:20.488935    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78180f7_be5e_48dd_90e3_fb497aa53b50.slice/crio-ad4e5b31e6774595d9127e3ad5cc8b800b3dc5cf0deea6e10bfc36a62a83580d WatchSource:0}: Error finding container ad4e5b31e6774595d9127e3ad5cc8b800b3dc5cf0deea6e10bfc36a62a83580d: Status 404 returned error can't find the container with id ad4e5b31e6774595d9127e3ad5cc8b800b3dc5cf0deea6e10bfc36a62a83580d
Sep 06 10:38:21.056776 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:21.056745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg" event={"ID":"c78180f7-be5e-48dd-90e3-fb497aa53b50","Type":"ContainerStarted","Data":"97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027"}
Sep 06 10:38:21.057143 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:21.056795    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg" event={"ID":"c78180f7-be5e-48dd-90e3-fb497aa53b50","Type":"ContainerStarted","Data":"ad4e5b31e6774595d9127e3ad5cc8b800b3dc5cf0deea6e10bfc36a62a83580d"}
Sep 06 10:38:21.057143 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:21.056875    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:38:21.075061 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:21.075009    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg" podStartSLOduration=2.074992388 podStartE2EDuration="2.074992388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:21.073144905 +0000 UTC m=+989.556645038" watchObservedRunningTime="2026-09-06 10:38:21.074992388 +0000 UTC m=+989.558492494"
Sep 06 10:38:27.075566 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.075529    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"]
Sep 06 10:38:27.079019 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.079000    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.082827 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.082804    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 10:38:27.083172 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.083149    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 10:38:27.083422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.083404    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-965kn\""
Sep 06 10:38:27.083633 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.083615    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6751d7-serving-cert\""
Sep 06 10:38:27.083755 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.083707    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 10:38:27.090996 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.090931    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"]
Sep 06 10:38:27.145954 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.145919    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.146136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.146019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.146136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.146072    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xc4z\" (UniqueName: \"kubernetes.io/projected/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-kube-api-access-2xc4z\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.146136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.146097    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.247394 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.247361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xc4z\" (UniqueName: \"kubernetes.io/projected/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-kube-api-access-2xc4z\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.247570 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.247405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.247570 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.247464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.247570 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.247509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.248225 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.248194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.249682 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.249659    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.249851 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.249831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.254991 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.254969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xc4z\" (UniqueName: \"kubernetes.io/projected/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-kube-api-access-2xc4z\") pod \"maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.392099 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.391999    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:27.686319 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.686293    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"]
Sep 06 10:38:27.688815 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:38:27.688781    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fac2a2c_bc1e_4954_bdc7_8c5954143d2a.slice/crio-ff89892e7aef2c78072659ba36840ba08d263da15230644b3af1df02498809a2 WatchSource:0}: Error finding container ff89892e7aef2c78072659ba36840ba08d263da15230644b3af1df02498809a2: Status 404 returned error can't find the container with id ff89892e7aef2c78072659ba36840ba08d263da15230644b3af1df02498809a2
Sep 06 10:38:27.690243 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.690203    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:38:27.731258 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.725427    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"]
Sep 06 10:38:27.736045 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.736019    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"]
Sep 06 10:38:27.736195 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.736111    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:38:27.738588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.738565    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-6751d7\""
Sep 06 10:38:27.738710 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.738650    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-6751d7-dockercfg-468vt\""
Sep 06 10:38:27.752098 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.752078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56x79\" (UniqueName: \"kubernetes.io/projected/37ae96c9-a528-499e-98f5-a805cc2ceb6b-kube-api-access-56x79\") pod \"payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf\" (UID: \"37ae96c9-a528-499e-98f5-a805cc2ceb6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:38:27.752272 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.752129    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37ae96c9-a528-499e-98f5-a805cc2ceb6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf\" (UID: \"37ae96c9-a528-499e-98f5-a805cc2ceb6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:38:27.853496 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.853464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37ae96c9-a528-499e-98f5-a805cc2ceb6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf\" (UID: \"37ae96c9-a528-499e-98f5-a805cc2ceb6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:38:27.853668 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.853549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56x79\" (UniqueName: \"kubernetes.io/projected/37ae96c9-a528-499e-98f5-a805cc2ceb6b-kube-api-access-56x79\") pod \"payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf\" (UID: \"37ae96c9-a528-499e-98f5-a805cc2ceb6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:38:27.854131 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.854109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37ae96c9-a528-499e-98f5-a805cc2ceb6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf\" (UID: \"37ae96c9-a528-499e-98f5-a805cc2ceb6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:38:27.862069 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:27.862048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56x79\" (UniqueName: \"kubernetes.io/projected/37ae96c9-a528-499e-98f5-a805cc2ceb6b-kube-api-access-56x79\") pod \"payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf\" (UID: \"37ae96c9-a528-499e-98f5-a805cc2ceb6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:38:28.049813 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:28.049785    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:38:28.096129 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:28.096049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx" event={"ID":"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a","Type":"ContainerStarted","Data":"ff89892e7aef2c78072659ba36840ba08d263da15230644b3af1df02498809a2"}
Sep 06 10:38:28.321817 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:28.321721    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"]
Sep 06 10:38:28.328622 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:38:28.328574    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ae96c9_a528_499e_98f5_a805cc2ceb6b.slice/crio-2528387ed87a2f324af0fb6634ef48f4157a9af1a71fc32f5ebe6a325448a5a9 WatchSource:0}: Error finding container 2528387ed87a2f324af0fb6634ef48f4157a9af1a71fc32f5ebe6a325448a5a9: Status 404 returned error can't find the container with id 2528387ed87a2f324af0fb6634ef48f4157a9af1a71fc32f5ebe6a325448a5a9
Sep 06 10:38:29.102655 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:29.102541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx" event={"ID":"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a","Type":"ContainerStarted","Data":"fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321"}
Sep 06 10:38:29.102655 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:29.102604    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:29.104392 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:29.104358    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf" event={"ID":"37ae96c9-a528-499e-98f5-a805cc2ceb6b","Type":"ContainerStarted","Data":"4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb"}
Sep 06 10:38:29.104551 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:29.104418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf" event={"ID":"37ae96c9-a528-499e-98f5-a805cc2ceb6b","Type":"ContainerStarted","Data":"2528387ed87a2f324af0fb6634ef48f4157a9af1a71fc32f5ebe6a325448a5a9"}
Sep 06 10:38:29.104652 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:29.104628    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:38:29.120864 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:29.120799    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx" podStartSLOduration=1.70384373 podStartE2EDuration="2.120777169s" totalImagesPullingTime="416.933439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:29.119652628 +0000 UTC m=+997.603152738" watchObservedRunningTime="2026-09-06 10:38:29.120777169 +0000 UTC m=+997.604277280"
Sep 06 10:38:29.136638 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:29.136582    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf" podStartSLOduration=2.136563051 podStartE2EDuration="2.136563051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:29.135448649 +0000 UTC m=+997.618948757" watchObservedRunningTime="2026-09-06 10:38:29.136563051 +0000 UTC m=+997.620063159"
Sep 06 10:38:32.064163 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:32.064128    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:38:32.548181 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:32.548148    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:51894->10.134.0.58:15021: read: connection reset by peer" start-of-body=
Sep 06 10:38:32.548391 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:32.548236    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:51894->10.134.0.58:15021: read: connection reset by peer"
Sep 06 10:38:33.127333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:33.127301    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerID="42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3" exitCode=0
Sep 06 10:38:33.127704 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:33.127349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" event={"ID":"4a86f003-7ddc-4a8e-889d-bdcd3ed11239","Type":"ContainerDied","Data":"42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3"}
Sep 06 10:38:33.127704 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:33.127700    2564 scope.go:117] "RemoveContainer" containerID="42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3"
Sep 06 10:38:33.130123 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:33.129331    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:38:33.130123 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:33.129511    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:38:33.130123 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:33.129606    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 10:38:34.135021 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:34.134986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" event={"ID":"4a86f003-7ddc-4a8e-889d-bdcd3ed11239","Type":"ContainerStarted","Data":"7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46"}
Sep 06 10:38:34.835487 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:34.835442    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:34.835733 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:34.835708    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 06 10:38:34.835788 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:34.835759    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 06 10:38:35.123555 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:35.123466    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:38:35.835574 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:35.835538    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 06 10:38:35.835974 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:35.835597    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 06 10:38:36.834914 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:36.834880    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 06 10:38:36.835121 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:36.834958    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 06 10:38:37.834932 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:37.834889    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 06 10:38:37.835405 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:37.834993    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 06 10:38:38.835861 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:38.835815    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 06 10:38:38.836370 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:38.835882    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 06 10:38:39.839132 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:39.839095    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:39.839533 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:39.839489    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:39.840254 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:39.840234    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:38:40.112812 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:40.112723    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:38:48.685778 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.685741    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"]
Sep 06 10:38:48.691206 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.691182    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.693708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.693687    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-71e210-serving-cert\""
Sep 06 10:38:48.701062 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.701041    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"]
Sep 06 10:38:48.752990 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.752959    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.753135 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.752999    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.753135 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.753033    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad9e14e-3d92-40be-a208-17f29bf06eb2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.753239 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.753188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g598q\" (UniqueName: \"kubernetes.io/projected/8ad9e14e-3d92-40be-a208-17f29bf06eb2-kube-api-access-g598q\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.854733 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.854702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.854904 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.854743    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.854904 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.854851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad9e14e-3d92-40be-a208-17f29bf06eb2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.854904 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:38:48.854856    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-serving-cert: secret "maas-api-e2e-shared-b-w2-71e210-serving-cert" not found
Sep 06 10:38:48.855076 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.854914    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g598q\" (UniqueName: \"kubernetes.io/projected/8ad9e14e-3d92-40be-a208-17f29bf06eb2-kube-api-access-g598q\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.855076 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:38:48.854945    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls podName:8ad9e14e-3d92-40be-a208-17f29bf06eb2 nodeName:}" failed. No retries permitted until 2026-09-06 10:38:49.354916161 +0000 UTC m=+1017.838416265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls") pod "maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f" (UID: "8ad9e14e-3d92-40be-a208-17f29bf06eb2") : secret "maas-api-e2e-shared-b-w2-71e210-serving-cert" not found
Sep 06 10:38:48.855578 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.855555    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad9e14e-3d92-40be-a208-17f29bf06eb2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.857155 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.857134    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:48.863224 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:48.863192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g598q\" (UniqueName: \"kubernetes.io/projected/8ad9e14e-3d92-40be-a208-17f29bf06eb2-kube-api-access-g598q\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:49.359600 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:49.359562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:49.359763 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:38:49.359708    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-serving-cert: secret "maas-api-e2e-shared-b-w2-71e210-serving-cert" not found
Sep 06 10:38:49.359807 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:38:49.359773    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls podName:8ad9e14e-3d92-40be-a208-17f29bf06eb2 nodeName:}" failed. No retries permitted until 2026-09-06 10:38:50.359756831 +0000 UTC m=+1018.843256918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls") pod "maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f" (UID: "8ad9e14e-3d92-40be-a208-17f29bf06eb2") : secret "maas-api-e2e-shared-b-w2-71e210-serving-cert" not found
Sep 06 10:38:50.370852 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:50.370823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:50.373182 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:50.373162    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:50.505302 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:50.505267    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:50.775833 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:50.775808    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"]
Sep 06 10:38:50.778305 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:38:50.778273    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad9e14e_3d92_40be_a208_17f29bf06eb2.slice/crio-9349ea35843f25fc7546fde54aeb9c73752a164b8da00cfa563f98927f6c910f WatchSource:0}: Error finding container 9349ea35843f25fc7546fde54aeb9c73752a164b8da00cfa563f98927f6c910f: Status 404 returned error can't find the container with id 9349ea35843f25fc7546fde54aeb9c73752a164b8da00cfa563f98927f6c910f
Sep 06 10:38:51.222852 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:51.222813    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f" event={"ID":"8ad9e14e-3d92-40be-a208-17f29bf06eb2","Type":"ContainerStarted","Data":"ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99"}
Sep 06 10:38:51.223026 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:51.222868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f" event={"ID":"8ad9e14e-3d92-40be-a208-17f29bf06eb2","Type":"ContainerStarted","Data":"9349ea35843f25fc7546fde54aeb9c73752a164b8da00cfa563f98927f6c910f"}
Sep 06 10:38:51.223026 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:51.222966    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:51.239720 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:51.239666    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f" podStartSLOduration=2.939326791 podStartE2EDuration="3.239647239s" totalImagesPullingTime="300.320448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:51.239139056 +0000 UTC m=+1019.722639179" watchObservedRunningTime="2026-09-06 10:38:51.239647239 +0000 UTC m=+1019.723147349"
Sep 06 10:38:56.281333 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.281298    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"]
Sep 06 10:38:56.286560 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.286536    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.289038 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.289018    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-682ea0-serving-cert\""
Sep 06 10:38:56.295510 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.295484    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"]
Sep 06 10:38:56.430653 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.430622    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.430832 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.430668    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6j2w\" (UniqueName: \"kubernetes.io/projected/1aeff591-42ec-419b-b7cb-bd18d25e0415-kube-api-access-s6j2w\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.430832 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.430694    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeff591-42ec-419b-b7cb-bd18d25e0415-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.430832 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.430768    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.531676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.531593    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6j2w\" (UniqueName: \"kubernetes.io/projected/1aeff591-42ec-419b-b7cb-bd18d25e0415-kube-api-access-s6j2w\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.531676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.531637    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeff591-42ec-419b-b7cb-bd18d25e0415-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.531871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.531695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.531871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.531800    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.532267 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.532237    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeff591-42ec-419b-b7cb-bd18d25e0415-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.534435 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.534411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.534435 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.534423    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.539516 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.539493    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6j2w\" (UniqueName: \"kubernetes.io/projected/1aeff591-42ec-419b-b7cb-bd18d25e0415-kube-api-access-s6j2w\") pod \"maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.600710 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.600675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:56.877557 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:56.877534    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"]
Sep 06 10:38:56.880129 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:38:56.880099    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aeff591_42ec_419b_b7cb_bd18d25e0415.slice/crio-d448a2c3fbd48fab8f16d257d8c6a3923869a637a114b45ec159e17af660af17 WatchSource:0}: Error finding container d448a2c3fbd48fab8f16d257d8c6a3923869a637a114b45ec159e17af660af17: Status 404 returned error can't find the container with id d448a2c3fbd48fab8f16d257d8c6a3923869a637a114b45ec159e17af660af17
Sep 06 10:38:57.232871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:57.232798    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:38:57.253448 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:57.253418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz" event={"ID":"1aeff591-42ec-419b-b7cb-bd18d25e0415","Type":"ContainerStarted","Data":"d448a2c3fbd48fab8f16d257d8c6a3923869a637a114b45ec159e17af660af17"}
Sep 06 10:38:58.259554 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:58.259526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz" event={"ID":"1aeff591-42ec-419b-b7cb-bd18d25e0415","Type":"ContainerStarted","Data":"4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580"}
Sep 06 10:38:58.259973 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:58.259656    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:38:58.275916 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:38:58.275871    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz" podStartSLOduration=1.878160211 podStartE2EDuration="2.275856015s" totalImagesPullingTime="397.695804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:58.275531978 +0000 UTC m=+1026.759032105" watchObservedRunningTime="2026-09-06 10:38:58.275856015 +0000 UTC m=+1026.759356123"
Sep 06 10:39:04.273397 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:04.273367    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:39:31.414131 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.414095    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"]
Sep 06 10:39:31.419738 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.419708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.422152 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.422131    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b21e0db6-serving-cert\""
Sep 06 10:39:31.427371 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.427347    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"]
Sep 06 10:39:31.565008 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.564965    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-maas-api-tls\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.565243 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.565144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db06fa5b-dcd7-4cbe-b05d-69679b645686-openshift-service-ca\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.565243 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.565205    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ccrl\" (UniqueName: \"kubernetes.io/projected/db06fa5b-dcd7-4cbe-b05d-69679b645686-kube-api-access-8ccrl\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.565374 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.565274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-metrics-certs\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.666677 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.666581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db06fa5b-dcd7-4cbe-b05d-69679b645686-openshift-service-ca\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.666677 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.666638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ccrl\" (UniqueName: \"kubernetes.io/projected/db06fa5b-dcd7-4cbe-b05d-69679b645686-kube-api-access-8ccrl\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.666677 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.666671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-metrics-certs\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.666945 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.666717    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-maas-api-tls\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.667421 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.667386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db06fa5b-dcd7-4cbe-b05d-69679b645686-openshift-service-ca\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.669091 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.669066    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-maas-api-tls\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.669197 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.669119    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-metrics-certs\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.675569 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.675550    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ccrl\" (UniqueName: \"kubernetes.io/projected/db06fa5b-dcd7-4cbe-b05d-69679b645686-kube-api-access-8ccrl\") pod \"maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:31.735154 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:31.735115    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:32.009429 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.009401    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"]
Sep 06 10:39:32.011261 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:39:32.011237    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb06fa5b_dcd7_4cbe_b05d_69679b645686.slice/crio-f5668efffccd3359301925c39b74648730a20c92be1399d54f157afcb65fce3f WatchSource:0}: Error finding container f5668efffccd3359301925c39b74648730a20c92be1399d54f157afcb65fce3f: Status 404 returned error can't find the container with id f5668efffccd3359301925c39b74648730a20c92be1399d54f157afcb65fce3f
Sep 06 10:39:32.057965 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.057933    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"]
Sep 06 10:39:32.062763 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.062737    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:39:32.065108 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.065089    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b21e0db6\""
Sep 06 10:39:32.072642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.072617    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"]
Sep 06 10:39:32.172008 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.171972    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd03a45-ec2b-4182-9b55-af69f13f9335-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t\" (UID: \"efd03a45-ec2b-4182-9b55-af69f13f9335\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:39:32.172199 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.172071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fxjn\" (UniqueName: \"kubernetes.io/projected/efd03a45-ec2b-4182-9b55-af69f13f9335-kube-api-access-7fxjn\") pod \"payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t\" (UID: \"efd03a45-ec2b-4182-9b55-af69f13f9335\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:39:32.273675 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.273594    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd03a45-ec2b-4182-9b55-af69f13f9335-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t\" (UID: \"efd03a45-ec2b-4182-9b55-af69f13f9335\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:39:32.273675 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.273663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fxjn\" (UniqueName: \"kubernetes.io/projected/efd03a45-ec2b-4182-9b55-af69f13f9335-kube-api-access-7fxjn\") pod \"payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t\" (UID: \"efd03a45-ec2b-4182-9b55-af69f13f9335\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:39:32.274300 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.274280    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd03a45-ec2b-4182-9b55-af69f13f9335-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t\" (UID: \"efd03a45-ec2b-4182-9b55-af69f13f9335\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:39:32.282398 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.282377    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fxjn\" (UniqueName: \"kubernetes.io/projected/efd03a45-ec2b-4182-9b55-af69f13f9335-kube-api-access-7fxjn\") pod \"payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t\" (UID: \"efd03a45-ec2b-4182-9b55-af69f13f9335\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:39:32.374669 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.374630    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:39:32.445149 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.445114    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf" event={"ID":"db06fa5b-dcd7-4cbe-b05d-69679b645686","Type":"ContainerStarted","Data":"f5668efffccd3359301925c39b74648730a20c92be1399d54f157afcb65fce3f"}
Sep 06 10:39:32.673054 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:32.673027    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"]
Sep 06 10:39:32.675717 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:39:32.675678    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd03a45_ec2b_4182_9b55_af69f13f9335.slice/crio-d958c49002aba0593b28afdce97354b7eba2b4082597301f967bf000975cdf22 WatchSource:0}: Error finding container d958c49002aba0593b28afdce97354b7eba2b4082597301f967bf000975cdf22: Status 404 returned error can't find the container with id d958c49002aba0593b28afdce97354b7eba2b4082597301f967bf000975cdf22
Sep 06 10:39:33.452798 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:33.452756    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf" event={"ID":"db06fa5b-dcd7-4cbe-b05d-69679b645686","Type":"ContainerStarted","Data":"5cffc5b0a9b745a290a30f1d27db5a08bba4161398aebe070b261425dafbd5c3"}
Sep 06 10:39:33.453296 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:33.452815    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:33.454332 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:33.454307    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t" event={"ID":"efd03a45-ec2b-4182-9b55-af69f13f9335","Type":"ContainerStarted","Data":"3d64fa660c652d18fa4a5dec792c4632b41fc8a2c4948fd4ba009cb2efedfa54"}
Sep 06 10:39:33.454449 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:33.454354    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t" event={"ID":"efd03a45-ec2b-4182-9b55-af69f13f9335","Type":"ContainerStarted","Data":"d958c49002aba0593b28afdce97354b7eba2b4082597301f967bf000975cdf22"}
Sep 06 10:39:33.454449 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:33.454425    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:39:33.473970 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:33.473924    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf" podStartSLOduration=2.09085587 podStartE2EDuration="2.473909833s" totalImagesPullingTime="383.053963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:39:33.472575766 +0000 UTC m=+1061.956075895" watchObservedRunningTime="2026-09-06 10:39:33.473909833 +0000 UTC m=+1061.957409942"
Sep 06 10:39:33.487832 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:33.487784    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t" podStartSLOduration=1.4877692599999999 podStartE2EDuration="1.48776926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:39:33.48761593 +0000 UTC m=+1061.971116064" watchObservedRunningTime="2026-09-06 10:39:33.48776926 +0000 UTC m=+1061.971269369"
Sep 06 10:39:39.466159 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:39.466132    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:39:44.305001 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.304930    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"]
Sep 06 10:39:44.308660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.308641    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.312488 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.312429    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"default-dockercfg-ktwrx\""
Sep 06 10:39:44.312488 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.312429    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"kube-root-ca.crt\""
Sep 06 10:39:44.312757 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.312433    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"openshift-service-ca.crt\""
Sep 06 10:39:44.312757 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.312463    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"auth-test-model-6751d7-kserve-self-signed-certs\""
Sep 06 10:39:44.319090 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.319068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"]
Sep 06 10:39:44.391953 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.391919    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-home\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.392087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.391982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-model-cache\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.392087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.392039    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc6vl\" (UniqueName: \"kubernetes.io/projected/aa3fdae8-64f5-47db-867e-a50699103a57-kube-api-access-nc6vl\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.392087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.392060    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3fdae8-64f5-47db-867e-a50699103a57-tls-certs\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.392087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.392076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-dshm\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.392273 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.392090    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-tmp-dir\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.462291 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.462266    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:39:44.493143 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.493111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc6vl\" (UniqueName: \"kubernetes.io/projected/aa3fdae8-64f5-47db-867e-a50699103a57-kube-api-access-nc6vl\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.493331 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.493165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3fdae8-64f5-47db-867e-a50699103a57-tls-certs\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.493331 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.493223    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-dshm\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.493331 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.493253    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-tmp-dir\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.493510 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.493376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-home\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.493510 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.493480    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-model-cache\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.493740 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.493713    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-tmp-dir\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.493825 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.493806    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-model-cache\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.494113 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.494087    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-home\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.495551 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.495524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-dshm\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.495924 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.495902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3fdae8-64f5-47db-867e-a50699103a57-tls-certs\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.502084 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.502064    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc6vl\" (UniqueName: \"kubernetes.io/projected/aa3fdae8-64f5-47db-867e-a50699103a57-kube-api-access-nc6vl\") pod \"auth-test-model-6751d7-kserve-7c5b8547c5-rtwms\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.620623 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.620539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:44.893186 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:44.893160    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"]
Sep 06 10:39:44.895301 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:39:44.895267    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa3fdae8_64f5_47db_867e_a50699103a57.slice/crio-53bb05d052fd4ccc4e86bc89d4f301a7e65910bf0e10b5f1f5f69665e0b3f4d0 WatchSource:0}: Error finding container 53bb05d052fd4ccc4e86bc89d4f301a7e65910bf0e10b5f1f5f69665e0b3f4d0: Status 404 returned error can't find the container with id 53bb05d052fd4ccc4e86bc89d4f301a7e65910bf0e10b5f1f5f69665e0b3f4d0
Sep 06 10:39:45.517699 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:45.517665    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms" event={"ID":"aa3fdae8-64f5-47db-867e-a50699103a57","Type":"ContainerStarted","Data":"e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174"}
Sep 06 10:39:45.518131 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:45.517708    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms" event={"ID":"aa3fdae8-64f5-47db-867e-a50699103a57","Type":"ContainerStarted","Data":"53bb05d052fd4ccc4e86bc89d4f301a7e65910bf0e10b5f1f5f69665e0b3f4d0"}
Sep 06 10:39:45.534009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:45.533944    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms" podStartSLOduration=1.533930377 podStartE2EDuration="1.533930377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:39:45.532884264 +0000 UTC m=+1074.016384394" watchObservedRunningTime="2026-09-06 10:39:45.533930377 +0000 UTC m=+1074.017430487"
Sep 06 10:39:52.121994 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:52.121956    2564 scope.go:117] "RemoveContainer" containerID="494f1215c68ff0554ce70c3c42759399493a1e92760717894e6e9f97020b2555"
Sep 06 10:39:54.620962 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:54.620932    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:54.621395 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:54.620981    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:54.633388 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:54.633363    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:39:55.580354 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:39:55.580320    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:41:05.619540 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:05.619506    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"]
Sep 06 10:41:05.620136 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:05.619722    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf" podUID="db06fa5b-dcd7-4cbe-b05d-69679b645686" containerName="maas-api" containerID="cri-o://5cffc5b0a9b745a290a30f1d27db5a08bba4161398aebe070b261425dafbd5c3" gracePeriod=30
Sep 06 10:41:05.711972 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:05.711941    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"]
Sep 06 10:41:05.712259 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:05.712196    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t" podUID="efd03a45-ec2b-4182-9b55-af69f13f9335" containerName="payload-pre-processing" containerID="cri-o://3d64fa660c652d18fa4a5dec792c4632b41fc8a2c4948fd4ba009cb2efedfa54" gracePeriod=30
Sep 06 10:41:05.771942 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:41:05.771910    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd03a45_ec2b_4182_9b55_af69f13f9335.slice/crio-conmon-3d64fa660c652d18fa4a5dec792c4632b41fc8a2c4948fd4ba009cb2efedfa54.scope\": RecentStats: unable to find data in memory cache]"
Sep 06 10:41:05.924888 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:05.924853    2564 generic.go:356] "Generic (PLEG): container finished" podID="db06fa5b-dcd7-4cbe-b05d-69679b645686" containerID="5cffc5b0a9b745a290a30f1d27db5a08bba4161398aebe070b261425dafbd5c3" exitCode=0
Sep 06 10:41:05.925030 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:05.924908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf" event={"ID":"db06fa5b-dcd7-4cbe-b05d-69679b645686","Type":"ContainerDied","Data":"5cffc5b0a9b745a290a30f1d27db5a08bba4161398aebe070b261425dafbd5c3"}
Sep 06 10:41:05.926756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:05.926730    2564 generic.go:356] "Generic (PLEG): container finished" podID="efd03a45-ec2b-4182-9b55-af69f13f9335" containerID="3d64fa660c652d18fa4a5dec792c4632b41fc8a2c4948fd4ba009cb2efedfa54" exitCode=0
Sep 06 10:41:05.926858 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:05.926768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t" event={"ID":"efd03a45-ec2b-4182-9b55-af69f13f9335","Type":"ContainerDied","Data":"3d64fa660c652d18fa4a5dec792c4632b41fc8a2c4948fd4ba009cb2efedfa54"}
Sep 06 10:41:05.957692 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:05.957671    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:41:05.986263 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:05.986240    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:41:06.077541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.077515    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db06fa5b-dcd7-4cbe-b05d-69679b645686-openshift-service-ca\") pod \"db06fa5b-dcd7-4cbe-b05d-69679b645686\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") "
Sep 06 10:41:06.077541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.077545    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fxjn\" (UniqueName: \"kubernetes.io/projected/efd03a45-ec2b-4182-9b55-af69f13f9335-kube-api-access-7fxjn\") pod \"efd03a45-ec2b-4182-9b55-af69f13f9335\" (UID: \"efd03a45-ec2b-4182-9b55-af69f13f9335\") "
Sep 06 10:41:06.077764 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.077565    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-maas-api-tls\") pod \"db06fa5b-dcd7-4cbe-b05d-69679b645686\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") "
Sep 06 10:41:06.077764 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.077620    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd03a45-ec2b-4182-9b55-af69f13f9335-plugins-config-volume\") pod \"efd03a45-ec2b-4182-9b55-af69f13f9335\" (UID: \"efd03a45-ec2b-4182-9b55-af69f13f9335\") "
Sep 06 10:41:06.077764 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.077659    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ccrl\" (UniqueName: \"kubernetes.io/projected/db06fa5b-dcd7-4cbe-b05d-69679b645686-kube-api-access-8ccrl\") pod \"db06fa5b-dcd7-4cbe-b05d-69679b645686\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") "
Sep 06 10:41:06.077764 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.077687    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-metrics-certs\") pod \"db06fa5b-dcd7-4cbe-b05d-69679b645686\" (UID: \"db06fa5b-dcd7-4cbe-b05d-69679b645686\") "
Sep 06 10:41:06.077972 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.077913    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db06fa5b-dcd7-4cbe-b05d-69679b645686-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db06fa5b-dcd7-4cbe-b05d-69679b645686" (UID: "db06fa5b-dcd7-4cbe-b05d-69679b645686"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:41:06.078066 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.078040    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd03a45-ec2b-4182-9b55-af69f13f9335-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "efd03a45-ec2b-4182-9b55-af69f13f9335" (UID: "efd03a45-ec2b-4182-9b55-af69f13f9335"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:41:06.078133 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.078113    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db06fa5b-dcd7-4cbe-b05d-69679b645686-openshift-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:41:06.079844 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.079820    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db06fa5b-dcd7-4cbe-b05d-69679b645686" (UID: "db06fa5b-dcd7-4cbe-b05d-69679b645686"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:41:06.079949 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.079823    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db06fa5b-dcd7-4cbe-b05d-69679b645686-kube-api-access-8ccrl" (OuterVolumeSpecName: "kube-api-access-8ccrl") pod "db06fa5b-dcd7-4cbe-b05d-69679b645686" (UID: "db06fa5b-dcd7-4cbe-b05d-69679b645686"). InnerVolumeSpecName "kube-api-access-8ccrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:41:06.080060 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.080042    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "db06fa5b-dcd7-4cbe-b05d-69679b645686" (UID: "db06fa5b-dcd7-4cbe-b05d-69679b645686"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:41:06.080060 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.080050    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd03a45-ec2b-4182-9b55-af69f13f9335-kube-api-access-7fxjn" (OuterVolumeSpecName: "kube-api-access-7fxjn") pod "efd03a45-ec2b-4182-9b55-af69f13f9335" (UID: "efd03a45-ec2b-4182-9b55-af69f13f9335"). InnerVolumeSpecName "kube-api-access-7fxjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:41:06.179251 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.179141    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-metrics-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:41:06.179251 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.179170    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fxjn\" (UniqueName: \"kubernetes.io/projected/efd03a45-ec2b-4182-9b55-af69f13f9335-kube-api-access-7fxjn\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:41:06.179251 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.179182    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db06fa5b-dcd7-4cbe-b05d-69679b645686-maas-api-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:41:06.179251 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.179191    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd03a45-ec2b-4182-9b55-af69f13f9335-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:41:06.179251 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.179201    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ccrl\" (UniqueName: \"kubernetes.io/projected/db06fa5b-dcd7-4cbe-b05d-69679b645686-kube-api-access-8ccrl\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:41:06.933867 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.933779    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"
Sep 06 10:41:06.933867 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.933813    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf" event={"ID":"db06fa5b-dcd7-4cbe-b05d-69679b645686","Type":"ContainerDied","Data":"f5668efffccd3359301925c39b74648730a20c92be1399d54f157afcb65fce3f"}
Sep 06 10:41:06.934379 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.933887    2564 scope.go:117] "RemoveContainer" containerID="5cffc5b0a9b745a290a30f1d27db5a08bba4161398aebe070b261425dafbd5c3"
Sep 06 10:41:06.935255 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.935130    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"
Sep 06 10:41:06.935255 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.935127    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t" event={"ID":"efd03a45-ec2b-4182-9b55-af69f13f9335","Type":"ContainerDied","Data":"d958c49002aba0593b28afdce97354b7eba2b4082597301f967bf000975cdf22"}
Sep 06 10:41:06.943126 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.943089    2564 scope.go:117] "RemoveContainer" containerID="3d64fa660c652d18fa4a5dec792c4632b41fc8a2c4948fd4ba009cb2efedfa54"
Sep 06 10:41:06.960272 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.960251    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"]
Sep 06 10:41:06.964573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.964550    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b21e0db6-647fffbf78-rfd6t"]
Sep 06 10:41:06.974411 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.974387    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"]
Sep 06 10:41:06.978032 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:06.978012    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21e0db6-d696f4599-zjmzf"]
Sep 06 10:41:08.028668 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:08.028632    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db06fa5b-dcd7-4cbe-b05d-69679b645686" path="/var/lib/kubelet/pods/db06fa5b-dcd7-4cbe-b05d-69679b645686/volumes"
Sep 06 10:41:08.029031 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:08.028977    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efd03a45-ec2b-4182-9b55-af69f13f9335" path="/var/lib/kubelet/pods/efd03a45-ec2b-4182-9b55-af69f13f9335/volumes"
Sep 06 10:41:52.013631 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:52.013599    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:41:52.017785 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:41:52.017764    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:42:42.159788 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:42.159757    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"]
Sep 06 10:42:56.380964 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.380931    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"]
Sep 06 10:42:56.381419 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.381385    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd03a45-ec2b-4182-9b55-af69f13f9335" containerName="payload-pre-processing"
Sep 06 10:42:56.381419 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.381399    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd03a45-ec2b-4182-9b55-af69f13f9335" containerName="payload-pre-processing"
Sep 06 10:42:56.381419 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.381420    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db06fa5b-dcd7-4cbe-b05d-69679b645686" containerName="maas-api"
Sep 06 10:42:56.381526 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.381426    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db06fa5b-dcd7-4cbe-b05d-69679b645686" containerName="maas-api"
Sep 06 10:42:56.381526 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.381498    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="db06fa5b-dcd7-4cbe-b05d-69679b645686" containerName="maas-api"
Sep 06 10:42:56.381526 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.381505    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="efd03a45-ec2b-4182-9b55-af69f13f9335" containerName="payload-pre-processing"
Sep 06 10:42:56.384346 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.384330    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:42:56.386852 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.386830    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-616a10\""
Sep 06 10:42:56.386966 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.386934    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-616a10-dockercfg-bkrzh\""
Sep 06 10:42:56.394686 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.394665    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"]
Sep 06 10:42:56.411281 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.411260    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"]
Sep 06 10:42:56.414596 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.414579    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:42:56.424472 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.424450    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"]
Sep 06 10:42:56.536676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.536644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7bl2\" (UniqueName: \"kubernetes.io/projected/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-kube-api-access-v7bl2\") pod \"payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph\" (UID: \"f8e2754f-136b-40d4-b99f-a8f25fd01ec9\") " pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:42:56.536884 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.536702    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68gkv\" (UniqueName: \"kubernetes.io/projected/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-kube-api-access-68gkv\") pod \"payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782\" (UID: \"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:42:56.536884 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.536797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph\" (UID: \"f8e2754f-136b-40d4-b99f-a8f25fd01ec9\") " pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:42:56.536884 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.536845    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782\" (UID: \"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:42:56.637763 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.637665    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7bl2\" (UniqueName: \"kubernetes.io/projected/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-kube-api-access-v7bl2\") pod \"payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph\" (UID: \"f8e2754f-136b-40d4-b99f-a8f25fd01ec9\") " pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:42:56.637763 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.637754    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gkv\" (UniqueName: \"kubernetes.io/projected/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-kube-api-access-68gkv\") pod \"payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782\" (UID: \"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:42:56.637986 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.637811    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph\" (UID: \"f8e2754f-136b-40d4-b99f-a8f25fd01ec9\") " pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:42:56.637986 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.637845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782\" (UID: \"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:42:56.638586 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.638558    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph\" (UID: \"f8e2754f-136b-40d4-b99f-a8f25fd01ec9\") " pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:42:56.638692 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.638583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782\" (UID: \"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:42:56.645523 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.645501    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68gkv\" (UniqueName: \"kubernetes.io/projected/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-kube-api-access-68gkv\") pod \"payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782\" (UID: \"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:42:56.645621 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.645567    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7bl2\" (UniqueName: \"kubernetes.io/projected/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-kube-api-access-v7bl2\") pod \"payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph\" (UID: \"f8e2754f-136b-40d4-b99f-a8f25fd01ec9\") " pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:42:56.695386 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.695359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:42:56.725111 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.725083    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:42:56.971499 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:56.971475    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"]
Sep 06 10:42:56.972982 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:42:56.972946    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c8d031_cb9d_4c1b_98d9_8a1ea03df93b.slice/crio-8a7ee5cab94daf2d1985adf3bce83183bc949a672f68c45c35abf85fdef2c789 WatchSource:0}: Error finding container 8a7ee5cab94daf2d1985adf3bce83183bc949a672f68c45c35abf85fdef2c789: Status 404 returned error can't find the container with id 8a7ee5cab94daf2d1985adf3bce83183bc949a672f68c45c35abf85fdef2c789
Sep 06 10:42:57.006589 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.006525    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"]
Sep 06 10:42:57.007464 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:42:57.007438    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e2754f_136b_40d4_b99f_a8f25fd01ec9.slice/crio-4cec73aed95fdb100ff411f4f57b866aff3bdb04d9cc9e422d856d4074bfe176 WatchSource:0}: Error finding container 4cec73aed95fdb100ff411f4f57b866aff3bdb04d9cc9e422d856d4074bfe176: Status 404 returned error can't find the container with id 4cec73aed95fdb100ff411f4f57b866aff3bdb04d9cc9e422d856d4074bfe176
Sep 06 10:42:57.179596 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.179558    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms" podUID="aa3fdae8-64f5-47db-867e-a50699103a57" containerName="main" containerID="cri-o://e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174" gracePeriod=45
Sep 06 10:42:57.419276 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.419248    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:42:57.490011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.489981    2564 generic.go:356] "Generic (PLEG): container finished" podID="aa3fdae8-64f5-47db-867e-a50699103a57" containerID="e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174" exitCode=0
Sep 06 10:42:57.490198 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.490021    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"
Sep 06 10:42:57.490198 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.490054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms" event={"ID":"aa3fdae8-64f5-47db-867e-a50699103a57","Type":"ContainerDied","Data":"e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174"}
Sep 06 10:42:57.490198 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.490091    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms" event={"ID":"aa3fdae8-64f5-47db-867e-a50699103a57","Type":"ContainerDied","Data":"53bb05d052fd4ccc4e86bc89d4f301a7e65910bf0e10b5f1f5f69665e0b3f4d0"}
Sep 06 10:42:57.490198 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.490112    2564 scope.go:117] "RemoveContainer" containerID="e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174"
Sep 06 10:42:57.491929 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.491901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782" event={"ID":"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b","Type":"ContainerStarted","Data":"675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024"}
Sep 06 10:42:57.492048 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.491950    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782" event={"ID":"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b","Type":"ContainerStarted","Data":"8a7ee5cab94daf2d1985adf3bce83183bc949a672f68c45c35abf85fdef2c789"}
Sep 06 10:42:57.492048 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.491997    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:42:57.493670 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.493648    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph" event={"ID":"f8e2754f-136b-40d4-b99f-a8f25fd01ec9","Type":"ContainerStarted","Data":"6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140"}
Sep 06 10:42:57.493790 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.493686    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph" event={"ID":"f8e2754f-136b-40d4-b99f-a8f25fd01ec9","Type":"ContainerStarted","Data":"4cec73aed95fdb100ff411f4f57b866aff3bdb04d9cc9e422d856d4074bfe176"}
Sep 06 10:42:57.493790 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.493766    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:42:57.498850 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.498827    2564 scope.go:117] "RemoveContainer" containerID="e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174"
Sep 06 10:42:57.499134 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:42:57.499115    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174\": container with ID starting with e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174 not found: ID does not exist" containerID="e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174"
Sep 06 10:42:57.499189 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.499150    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174"} err="failed to get container status \"e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174\": rpc error: code = NotFound desc = could not find container \"e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174\": container with ID starting with e58636ef6fc53cea92e864fd62b0c2a562bf12858f388913398b612e1e40c174 not found: ID does not exist"
Sep 06 10:42:57.516303 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.516264    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782" podStartSLOduration=1.5162529 podStartE2EDuration="1.5162529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:42:57.514269209 +0000 UTC m=+1265.997769343" watchObservedRunningTime="2026-09-06 10:42:57.5162529 +0000 UTC m=+1265.999753008"
Sep 06 10:42:57.533469 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.533427    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph" podStartSLOduration=1.5334155520000001 podStartE2EDuration="1.533415552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:42:57.53119459 +0000 UTC m=+1266.014694725" watchObservedRunningTime="2026-09-06 10:42:57.533415552 +0000 UTC m=+1266.016915660"
Sep 06 10:42:57.545519 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.545460    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-home\") pod \"aa3fdae8-64f5-47db-867e-a50699103a57\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") "
Sep 06 10:42:57.545519 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.545492    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3fdae8-64f5-47db-867e-a50699103a57-tls-certs\") pod \"aa3fdae8-64f5-47db-867e-a50699103a57\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") "
Sep 06 10:42:57.545519 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.545507    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-tmp-dir\") pod \"aa3fdae8-64f5-47db-867e-a50699103a57\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") "
Sep 06 10:42:57.545723 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.545547    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-dshm\") pod \"aa3fdae8-64f5-47db-867e-a50699103a57\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") "
Sep 06 10:42:57.545723 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.545570    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-model-cache\") pod \"aa3fdae8-64f5-47db-867e-a50699103a57\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") "
Sep 06 10:42:57.545723 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.545607    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc6vl\" (UniqueName: \"kubernetes.io/projected/aa3fdae8-64f5-47db-867e-a50699103a57-kube-api-access-nc6vl\") pod \"aa3fdae8-64f5-47db-867e-a50699103a57\" (UID: \"aa3fdae8-64f5-47db-867e-a50699103a57\") "
Sep 06 10:42:57.545886 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.545740    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-home" (OuterVolumeSpecName: "home") pod "aa3fdae8-64f5-47db-867e-a50699103a57" (UID: "aa3fdae8-64f5-47db-867e-a50699103a57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:42:57.545886 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.545772    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa3fdae8-64f5-47db-867e-a50699103a57" (UID: "aa3fdae8-64f5-47db-867e-a50699103a57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:42:57.545993 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.545955    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa3fdae8-64f5-47db-867e-a50699103a57" (UID: "aa3fdae8-64f5-47db-867e-a50699103a57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:42:57.546047 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.545992    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:42:57.546047 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.546012    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:42:57.547775 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.547752    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3fdae8-64f5-47db-867e-a50699103a57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa3fdae8-64f5-47db-867e-a50699103a57" (UID: "aa3fdae8-64f5-47db-867e-a50699103a57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:42:57.548112 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.548089    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-dshm" (OuterVolumeSpecName: "dshm") pod "aa3fdae8-64f5-47db-867e-a50699103a57" (UID: "aa3fdae8-64f5-47db-867e-a50699103a57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:42:57.548262 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.548127    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3fdae8-64f5-47db-867e-a50699103a57-kube-api-access-nc6vl" (OuterVolumeSpecName: "kube-api-access-nc6vl") pod "aa3fdae8-64f5-47db-867e-a50699103a57" (UID: "aa3fdae8-64f5-47db-867e-a50699103a57"). InnerVolumeSpecName "kube-api-access-nc6vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:42:57.647330 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.647223    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3fdae8-64f5-47db-867e-a50699103a57-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:42:57.647330 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.647253    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:42:57.647330 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.647267    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdae8-64f5-47db-867e-a50699103a57-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:42:57.647330 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.647280    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc6vl\" (UniqueName: \"kubernetes.io/projected/aa3fdae8-64f5-47db-867e-a50699103a57-kube-api-access-nc6vl\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:42:57.812762 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.812732    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"]
Sep 06 10:42:57.816708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:57.816682    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/auth-test-model-6751d7-kserve-7c5b8547c5-rtwms"]
Sep 06 10:42:58.030800 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:42:58.030765    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa3fdae8-64f5-47db-867e-a50699103a57" path="/var/lib/kubelet/pods/aa3fdae8-64f5-47db-867e-a50699103a57/volumes"
Sep 06 10:43:08.502414 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:08.502372    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:43:08.502928 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:08.502648    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:43:09.792718 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.792684    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"]
Sep 06 10:43:09.793117 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.793104    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3fdae8-64f5-47db-867e-a50699103a57" containerName="main"
Sep 06 10:43:09.793169 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.793119    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3fdae8-64f5-47db-867e-a50699103a57" containerName="main"
Sep 06 10:43:09.793227 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.793200    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3fdae8-64f5-47db-867e-a50699103a57" containerName="main"
Sep 06 10:43:09.797777 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.797753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.801447 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.801426    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"default-dockercfg-ktwrx\""
Sep 06 10:43:09.801447 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.801426    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"openshift-service-ca.crt\""
Sep 06 10:43:09.801447 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.801452    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"sub-test-model-6751d7-kserve-self-signed-certs\""
Sep 06 10:43:09.801676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.801500    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"kube-root-ca.crt\""
Sep 06 10:43:09.806343 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.806325    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"]
Sep 06 10:43:09.857972 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.857944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-dshm\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.858103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.857988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-home\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.858103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.858015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tls-certs\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.858103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.858069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nc2b\" (UniqueName: \"kubernetes.io/projected/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-kube-api-access-8nc2b\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.858421 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.858146    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-model-cache\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.858421 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.858181    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tmp-dir\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.959189 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.959157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-home\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.959363 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.959197    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tls-certs\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.959363 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.959292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nc2b\" (UniqueName: \"kubernetes.io/projected/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-kube-api-access-8nc2b\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.959456 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.959436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-model-cache\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.959523 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.959501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tmp-dir\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.959676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.959553    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-dshm\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.959676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.959620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-home\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.959824 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.959804    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tmp-dir\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.959824 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.959819    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-model-cache\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.961596 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.961573    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-dshm\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.961879 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.961854    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tls-certs\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:09.967317 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:09.967292    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nc2b\" (UniqueName: \"kubernetes.io/projected/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-kube-api-access-8nc2b\") pod \"sub-test-model-6751d7-kserve-9db966c6c-82vh7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:10.108149 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:10.108062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:10.380269 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:10.380245    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"]
Sep 06 10:43:10.382865 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:43:10.382838    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef49ab32_70f1_4768_ad94_3a4aa5314bc7.slice/crio-4d04a77ba6e19092584ce5b91dd7aac0f81ac6d8ee2ccf802e6e07a30163bc3e WatchSource:0}: Error finding container 4d04a77ba6e19092584ce5b91dd7aac0f81ac6d8ee2ccf802e6e07a30163bc3e: Status 404 returned error can't find the container with id 4d04a77ba6e19092584ce5b91dd7aac0f81ac6d8ee2ccf802e6e07a30163bc3e
Sep 06 10:43:10.568365 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:10.568325    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7" event={"ID":"ef49ab32-70f1-4768-ad94-3a4aa5314bc7","Type":"ContainerStarted","Data":"88fb7fd8fcbdf281c8168812ec45160acf70d7fd7f1c0bbef3424c285f5c48e3"}
Sep 06 10:43:10.568365 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:10.568369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7" event={"ID":"ef49ab32-70f1-4768-ad94-3a4aa5314bc7","Type":"ContainerStarted","Data":"4d04a77ba6e19092584ce5b91dd7aac0f81ac6d8ee2ccf802e6e07a30163bc3e"}
Sep 06 10:43:10.587584 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:10.587537    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7" podStartSLOduration=1.58752219 podStartE2EDuration="1.58752219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:43:10.585715515 +0000 UTC m=+1279.069215666" watchObservedRunningTime="2026-09-06 10:43:10.58752219 +0000 UTC m=+1279.071022299"
Sep 06 10:43:20.108448 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:20.108414    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:20.108886 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:20.108477    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:20.121234 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:20.121189    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:20.632721 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:20.632690    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:43:25.860491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:25.860433    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"]
Sep 06 10:43:25.863871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:25.863856    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:43:25.866553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:25.866534    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-40f309\""
Sep 06 10:43:25.866659 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:25.866542    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-40f309-dockercfg-x4w2f\""
Sep 06 10:43:25.875839 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:25.875818    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"]
Sep 06 10:43:26.019983 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.019954    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfcv7\" (UniqueName: \"kubernetes.io/projected/c44d6f55-dfe4-46a2-a7ba-15884230b38e-kube-api-access-tfcv7\") pod \"payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th\" (UID: \"c44d6f55-dfe4-46a2-a7ba-15884230b38e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:43:26.020151 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.020017    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44d6f55-dfe4-46a2-a7ba-15884230b38e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th\" (UID: \"c44d6f55-dfe4-46a2-a7ba-15884230b38e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:43:26.121370 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.121296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44d6f55-dfe4-46a2-a7ba-15884230b38e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th\" (UID: \"c44d6f55-dfe4-46a2-a7ba-15884230b38e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:43:26.121537 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.121392    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfcv7\" (UniqueName: \"kubernetes.io/projected/c44d6f55-dfe4-46a2-a7ba-15884230b38e-kube-api-access-tfcv7\") pod \"payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th\" (UID: \"c44d6f55-dfe4-46a2-a7ba-15884230b38e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:43:26.122034 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.122006    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44d6f55-dfe4-46a2-a7ba-15884230b38e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th\" (UID: \"c44d6f55-dfe4-46a2-a7ba-15884230b38e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:43:26.129630 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.129608    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfcv7\" (UniqueName: \"kubernetes.io/projected/c44d6f55-dfe4-46a2-a7ba-15884230b38e-kube-api-access-tfcv7\") pod \"payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th\" (UID: \"c44d6f55-dfe4-46a2-a7ba-15884230b38e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:43:26.175453 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.175417    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:43:26.443933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.443908    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"]
Sep 06 10:43:26.445659 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:43:26.445636    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44d6f55_dfe4_46a2_a7ba_15884230b38e.slice/crio-d25cf7ed8445b0234b8feccb0561da651ee0db8333db9eecde84f211fd772d9d WatchSource:0}: Error finding container d25cf7ed8445b0234b8feccb0561da651ee0db8333db9eecde84f211fd772d9d: Status 404 returned error can't find the container with id d25cf7ed8445b0234b8feccb0561da651ee0db8333db9eecde84f211fd772d9d
Sep 06 10:43:26.652912 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.652873    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th" event={"ID":"c44d6f55-dfe4-46a2-a7ba-15884230b38e","Type":"ContainerStarted","Data":"bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61"}
Sep 06 10:43:26.653104 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.652947    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th" event={"ID":"c44d6f55-dfe4-46a2-a7ba-15884230b38e","Type":"ContainerStarted","Data":"d25cf7ed8445b0234b8feccb0561da651ee0db8333db9eecde84f211fd772d9d"}
Sep 06 10:43:26.653104 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.653003    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:43:26.673509 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:26.673380    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th" podStartSLOduration=1.673362875 podStartE2EDuration="1.673362875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:43:26.67300443 +0000 UTC m=+1295.156504558" watchObservedRunningTime="2026-09-06 10:43:26.673362875 +0000 UTC m=+1295.156862986"
Sep 06 10:43:31.397844 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.397807    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"]
Sep 06 10:43:31.402677 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.402659    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.406188 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.406164    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"kube-root-ca.crt\""
Sep 06 10:43:31.406398 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.406182    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"openshift-service-ca.crt\""
Sep 06 10:43:31.406498 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.406193    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"default-dockercfg-p7bg2\""
Sep 06 10:43:31.406498 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.406193    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"sub-test-model-682ea0-kserve-self-signed-certs\""
Sep 06 10:43:31.409338 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.409318    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"]
Sep 06 10:43:31.575685 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.575647    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-tmp-dir\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.575685 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.575684    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-dshm\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.575908 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.575707    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpqzd\" (UniqueName: \"kubernetes.io/projected/5841eacf-26da-4a66-891a-d96f094f7e97-kube-api-access-cpqzd\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.575908 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.575735    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5841eacf-26da-4a66-891a-d96f094f7e97-tls-certs\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.575908 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.575843    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-home\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.575908 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.575879    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-model-cache\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.677237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.677127    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-tmp-dir\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.677237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.677160    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-dshm\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.677237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.677177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpqzd\" (UniqueName: \"kubernetes.io/projected/5841eacf-26da-4a66-891a-d96f094f7e97-kube-api-access-cpqzd\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.677237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.677195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5841eacf-26da-4a66-891a-d96f094f7e97-tls-certs\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.677570 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.677242    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-home\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.677570 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.677260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-model-cache\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.677665 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.677642    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-tmp-dir\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.677709 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.677667    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-home\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.677752 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.677716    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-model-cache\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.679594 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.679565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-dshm\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.679795 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.679774    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5841eacf-26da-4a66-891a-d96f094f7e97-tls-certs\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.687967 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.687944    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpqzd\" (UniqueName: \"kubernetes.io/projected/5841eacf-26da-4a66-891a-d96f094f7e97-kube-api-access-cpqzd\") pod \"sub-test-model-682ea0-kserve-74799f8d67-6cdh4\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.713746 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.713716    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:31.991452 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:31.987789    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"]
Sep 06 10:43:31.995069 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:43:31.995029    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5841eacf_26da_4a66_891a_d96f094f7e97.slice/crio-b1e6691daa6a345c66e6e1ded32412875e7cfda5d1b5e8c925db53c162f535bd WatchSource:0}: Error finding container b1e6691daa6a345c66e6e1ded32412875e7cfda5d1b5e8c925db53c162f535bd: Status 404 returned error can't find the container with id b1e6691daa6a345c66e6e1ded32412875e7cfda5d1b5e8c925db53c162f535bd
Sep 06 10:43:32.684025 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:32.683901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4" event={"ID":"5841eacf-26da-4a66-891a-d96f094f7e97","Type":"ContainerStarted","Data":"41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16"}
Sep 06 10:43:32.684025 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:32.683950    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4" event={"ID":"5841eacf-26da-4a66-891a-d96f094f7e97","Type":"ContainerStarted","Data":"b1e6691daa6a345c66e6e1ded32412875e7cfda5d1b5e8c925db53c162f535bd"}
Sep 06 10:43:32.702723 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:32.702679    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4" podStartSLOduration=1.7026648739999999 podStartE2EDuration="1.702664874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:43:32.700271351 +0000 UTC m=+1301.183771482" watchObservedRunningTime="2026-09-06 10:43:32.702664874 +0000 UTC m=+1301.186164982"
Sep 06 10:43:37.660151 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:37.660118    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:43:41.714562 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:41.714504    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:41.714562 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:41.714571    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:41.727549 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:41.727522    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:43:41.746299 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:43:41.746269    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:44:07.361880 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.361830    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"]
Sep 06 10:44:07.362337 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.362136    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph" podUID="f8e2754f-136b-40d4-b99f-a8f25fd01ec9" containerName="payload-processing" containerID="cri-o://6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140" gracePeriod=30
Sep 06 10:44:07.368048 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.368005    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"]
Sep 06 10:44:07.368319 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.368273    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782" podUID="00c8d031-cb9d-4c1b-98d9-8a1ea03df93b" containerName="payload-pre-processing" containerID="cri-o://675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024" gracePeriod=30
Sep 06 10:44:07.650673 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.650640    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:44:07.653938 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.653907    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:44:07.842141 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.842101    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-plugins-config-volume\") pod \"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b\" (UID: \"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b\") "
Sep 06 10:44:07.842395 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.842165    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-plugins-config-volume\") pod \"f8e2754f-136b-40d4-b99f-a8f25fd01ec9\" (UID: \"f8e2754f-136b-40d4-b99f-a8f25fd01ec9\") "
Sep 06 10:44:07.842395 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.842242    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7bl2\" (UniqueName: \"kubernetes.io/projected/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-kube-api-access-v7bl2\") pod \"f8e2754f-136b-40d4-b99f-a8f25fd01ec9\" (UID: \"f8e2754f-136b-40d4-b99f-a8f25fd01ec9\") "
Sep 06 10:44:07.842395 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.842274    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68gkv\" (UniqueName: \"kubernetes.io/projected/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-kube-api-access-68gkv\") pod \"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b\" (UID: \"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b\") "
Sep 06 10:44:07.842660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.842629    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00c8d031-cb9d-4c1b-98d9-8a1ea03df93b" (UID: "00c8d031-cb9d-4c1b-98d9-8a1ea03df93b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:07.842759 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.842712    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8e2754f-136b-40d4-b99f-a8f25fd01ec9" (UID: "f8e2754f-136b-40d4-b99f-a8f25fd01ec9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:07.844767 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.844732    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-kube-api-access-v7bl2" (OuterVolumeSpecName: "kube-api-access-v7bl2") pod "f8e2754f-136b-40d4-b99f-a8f25fd01ec9" (UID: "f8e2754f-136b-40d4-b99f-a8f25fd01ec9"). InnerVolumeSpecName "kube-api-access-v7bl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:07.844767 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.844746    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-kube-api-access-68gkv" (OuterVolumeSpecName: "kube-api-access-68gkv") pod "00c8d031-cb9d-4c1b-98d9-8a1ea03df93b" (UID: "00c8d031-cb9d-4c1b-98d9-8a1ea03df93b"). InnerVolumeSpecName "kube-api-access-68gkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:07.881001 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.880902    2564 generic.go:356] "Generic (PLEG): container finished" podID="f8e2754f-136b-40d4-b99f-a8f25fd01ec9" containerID="6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140" exitCode=0
Sep 06 10:44:07.881001 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.880955    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"
Sep 06 10:44:07.881001 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.880987    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph" event={"ID":"f8e2754f-136b-40d4-b99f-a8f25fd01ec9","Type":"ContainerDied","Data":"6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140"}
Sep 06 10:44:07.881321 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.881049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph" event={"ID":"f8e2754f-136b-40d4-b99f-a8f25fd01ec9","Type":"ContainerDied","Data":"4cec73aed95fdb100ff411f4f57b866aff3bdb04d9cc9e422d856d4074bfe176"}
Sep 06 10:44:07.881321 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.881084    2564 scope.go:117] "RemoveContainer" containerID="6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140"
Sep 06 10:44:07.882407 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.882371    2564 generic.go:356] "Generic (PLEG): container finished" podID="00c8d031-cb9d-4c1b-98d9-8a1ea03df93b" containerID="675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024" exitCode=0
Sep 06 10:44:07.882530 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.882415    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"
Sep 06 10:44:07.882530 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.882449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782" event={"ID":"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b","Type":"ContainerDied","Data":"675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024"}
Sep 06 10:44:07.882530 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.882509    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782" event={"ID":"00c8d031-cb9d-4c1b-98d9-8a1ea03df93b","Type":"ContainerDied","Data":"8a7ee5cab94daf2d1985adf3bce83183bc949a672f68c45c35abf85fdef2c789"}
Sep 06 10:44:07.890694 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.890665    2564 scope.go:117] "RemoveContainer" containerID="6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140"
Sep 06 10:44:07.891074 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:44:07.891039    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140\": container with ID starting with 6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140 not found: ID does not exist" containerID="6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140"
Sep 06 10:44:07.891183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.891085    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140"} err="failed to get container status \"6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140\": rpc error: code = NotFound desc = could not find container \"6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140\": container with ID starting with 6d5d38fbe3c25914932f9d1c9232631cfed1c90fec27e24accb6ac80bdb16140 not found: ID does not exist"
Sep 06 10:44:07.891183 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.891106    2564 scope.go:117] "RemoveContainer" containerID="675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024"
Sep 06 10:44:07.899959 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.899935    2564 scope.go:117] "RemoveContainer" containerID="675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024"
Sep 06 10:44:07.900360 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:44:07.900339    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024\": container with ID starting with 675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024 not found: ID does not exist" containerID="675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024"
Sep 06 10:44:07.900423 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.900371    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024"} err="failed to get container status \"675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024\": rpc error: code = NotFound desc = could not find container \"675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024\": container with ID starting with 675351d6bb65e5fa1c61316da80e69d91097e93ef7d738910abbb246d3a21024 not found: ID does not exist"
Sep 06 10:44:07.903792 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.903761    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"]
Sep 06 10:44:07.908194 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.908162    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616a10-79b9f5bd4f-vbhph"]
Sep 06 10:44:07.918762 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.918723    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"]
Sep 06 10:44:07.932381 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.932339    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616a10-6fcb8b69c8-7z782"]
Sep 06 10:44:07.943472 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.943434    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:07.943472 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.943469    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7bl2\" (UniqueName: \"kubernetes.io/projected/f8e2754f-136b-40d4-b99f-a8f25fd01ec9-kube-api-access-v7bl2\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:07.943684 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.943485    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68gkv\" (UniqueName: \"kubernetes.io/projected/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-kube-api-access-68gkv\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:07.943684 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:07.943498    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:08.029734 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:08.029686    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00c8d031-cb9d-4c1b-98d9-8a1ea03df93b" path="/var/lib/kubelet/pods/00c8d031-cb9d-4c1b-98d9-8a1ea03df93b/volumes"
Sep 06 10:44:08.030199 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:08.030177    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e2754f-136b-40d4-b99f-a8f25fd01ec9" path="/var/lib/kubelet/pods/f8e2754f-136b-40d4-b99f-a8f25fd01ec9/volumes"
Sep 06 10:44:23.872150 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:23.872061    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"]
Sep 06 10:44:25.706610 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:25.706574    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"]
Sep 06 10:44:30.056231 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:30.056174    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"]
Sep 06 10:44:30.056839 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:30.056438    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th" podUID="c44d6f55-dfe4-46a2-a7ba-15884230b38e" containerName="payload-pre-processing" containerID="cri-o://bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61" gracePeriod=30
Sep 06 10:44:30.323283 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:30.323180    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:44:30.369449 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:30.369413    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44d6f55-dfe4-46a2-a7ba-15884230b38e-plugins-config-volume\") pod \"c44d6f55-dfe4-46a2-a7ba-15884230b38e\" (UID: \"c44d6f55-dfe4-46a2-a7ba-15884230b38e\") "
Sep 06 10:44:30.369664 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:30.369526    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfcv7\" (UniqueName: \"kubernetes.io/projected/c44d6f55-dfe4-46a2-a7ba-15884230b38e-kube-api-access-tfcv7\") pod \"c44d6f55-dfe4-46a2-a7ba-15884230b38e\" (UID: \"c44d6f55-dfe4-46a2-a7ba-15884230b38e\") "
Sep 06 10:44:30.369926 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:30.369900    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44d6f55-dfe4-46a2-a7ba-15884230b38e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c44d6f55-dfe4-46a2-a7ba-15884230b38e" (UID: "c44d6f55-dfe4-46a2-a7ba-15884230b38e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:30.371678 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:30.371653    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44d6f55-dfe4-46a2-a7ba-15884230b38e-kube-api-access-tfcv7" (OuterVolumeSpecName: "kube-api-access-tfcv7") pod "c44d6f55-dfe4-46a2-a7ba-15884230b38e" (UID: "c44d6f55-dfe4-46a2-a7ba-15884230b38e"). InnerVolumeSpecName "kube-api-access-tfcv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:30.470607 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:30.470557    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfcv7\" (UniqueName: \"kubernetes.io/projected/c44d6f55-dfe4-46a2-a7ba-15884230b38e-kube-api-access-tfcv7\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:30.470607 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:30.470597    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44d6f55-dfe4-46a2-a7ba-15884230b38e-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:31.011898 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:31.011858    2564 generic.go:356] "Generic (PLEG): container finished" podID="c44d6f55-dfe4-46a2-a7ba-15884230b38e" containerID="bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61" exitCode=0
Sep 06 10:44:31.012087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:31.011908    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"
Sep 06 10:44:31.012087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:31.011948    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th" event={"ID":"c44d6f55-dfe4-46a2-a7ba-15884230b38e","Type":"ContainerDied","Data":"bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61"}
Sep 06 10:44:31.012087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:31.012009    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th" event={"ID":"c44d6f55-dfe4-46a2-a7ba-15884230b38e","Type":"ContainerDied","Data":"d25cf7ed8445b0234b8feccb0561da651ee0db8333db9eecde84f211fd772d9d"}
Sep 06 10:44:31.012087 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:31.012036    2564 scope.go:117] "RemoveContainer" containerID="bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61"
Sep 06 10:44:31.021961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:31.021941    2564 scope.go:117] "RemoveContainer" containerID="bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61"
Sep 06 10:44:31.022294 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:44:31.022274    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61\": container with ID starting with bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61 not found: ID does not exist" containerID="bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61"
Sep 06 10:44:31.022347 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:31.022304    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61"} err="failed to get container status \"bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61\": rpc error: code = NotFound desc = could not find container \"bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61\": container with ID starting with bd55ca47003ded3e2d3ea9f3aaafc89a17c5899198400d78b2ffba2481ac9d61 not found: ID does not exist"
Sep 06 10:44:31.034553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:31.034522    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"]
Sep 06 10:44:31.038275 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:31.038246    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40f309-57ff55b566-rt7th"]
Sep 06 10:44:32.035573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:32.035539    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c44d6f55-dfe4-46a2-a7ba-15884230b38e" path="/var/lib/kubelet/pods/c44d6f55-dfe4-46a2-a7ba-15884230b38e/volumes"
Sep 06 10:44:38.892188 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:38.892122    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7" podUID="ef49ab32-70f1-4768-ad94-3a4aa5314bc7" containerName="main" containerID="cri-o://88fb7fd8fcbdf281c8168812ec45160acf70d7fd7f1c0bbef3424c285f5c48e3" gracePeriod=45
Sep 06 10:44:39.064013 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.063981    2564 generic.go:356] "Generic (PLEG): container finished" podID="ef49ab32-70f1-4768-ad94-3a4aa5314bc7" containerID="88fb7fd8fcbdf281c8168812ec45160acf70d7fd7f1c0bbef3424c285f5c48e3" exitCode=0
Sep 06 10:44:39.064133 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.064048    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7" event={"ID":"ef49ab32-70f1-4768-ad94-3a4aa5314bc7","Type":"ContainerDied","Data":"88fb7fd8fcbdf281c8168812ec45160acf70d7fd7f1c0bbef3424c285f5c48e3"}
Sep 06 10:44:39.142811 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.142759    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:44:39.256287 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256257    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-dshm\") pod \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") "
Sep 06 10:44:39.256287 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256291    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-model-cache\") pod \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") "
Sep 06 10:44:39.256510 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256335    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tmp-dir\") pod \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") "
Sep 06 10:44:39.256510 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256358    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-home\") pod \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") "
Sep 06 10:44:39.256510 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256410    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tls-certs\") pod \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") "
Sep 06 10:44:39.256510 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256456    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nc2b\" (UniqueName: \"kubernetes.io/projected/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-kube-api-access-8nc2b\") pod \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\" (UID: \"ef49ab32-70f1-4768-ad94-3a4aa5314bc7\") "
Sep 06 10:44:39.256717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256619    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef49ab32-70f1-4768-ad94-3a4aa5314bc7" (UID: "ef49ab32-70f1-4768-ad94-3a4aa5314bc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:39.256717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256630    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-home" (OuterVolumeSpecName: "home") pod "ef49ab32-70f1-4768-ad94-3a4aa5314bc7" (UID: "ef49ab32-70f1-4768-ad94-3a4aa5314bc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:39.256717 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256647    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef49ab32-70f1-4768-ad94-3a4aa5314bc7" (UID: "ef49ab32-70f1-4768-ad94-3a4aa5314bc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:39.256864 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256848    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:39.256912 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256869    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:39.256912 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.256882    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:39.258198 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.258177    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-dshm" (OuterVolumeSpecName: "dshm") pod "ef49ab32-70f1-4768-ad94-3a4aa5314bc7" (UID: "ef49ab32-70f1-4768-ad94-3a4aa5314bc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:39.258443 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.258422    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-kube-api-access-8nc2b" (OuterVolumeSpecName: "kube-api-access-8nc2b") pod "ef49ab32-70f1-4768-ad94-3a4aa5314bc7" (UID: "ef49ab32-70f1-4768-ad94-3a4aa5314bc7"). InnerVolumeSpecName "kube-api-access-8nc2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:39.258510 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.258463    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef49ab32-70f1-4768-ad94-3a4aa5314bc7" (UID: "ef49ab32-70f1-4768-ad94-3a4aa5314bc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:39.357756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.357723    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:39.357756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.357753    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nc2b\" (UniqueName: \"kubernetes.io/projected/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-kube-api-access-8nc2b\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:39.357756 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:39.357764    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef49ab32-70f1-4768-ad94-3a4aa5314bc7-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:40.070293 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:40.070269    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"
Sep 06 10:44:40.070293 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:40.070281    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7" event={"ID":"ef49ab32-70f1-4768-ad94-3a4aa5314bc7","Type":"ContainerDied","Data":"4d04a77ba6e19092584ce5b91dd7aac0f81ac6d8ee2ccf802e6e07a30163bc3e"}
Sep 06 10:44:40.070726 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:40.070327    2564 scope.go:117] "RemoveContainer" containerID="88fb7fd8fcbdf281c8168812ec45160acf70d7fd7f1c0bbef3424c285f5c48e3"
Sep 06 10:44:40.088105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:40.088080    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"]
Sep 06 10:44:40.094304 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:40.094282    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/sub-test-model-6751d7-kserve-9db966c6c-82vh7"]
Sep 06 10:44:40.727111 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:40.727066    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4" podUID="5841eacf-26da-4a66-891a-d96f094f7e97" containerName="main" containerID="cri-o://41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16" gracePeriod=45
Sep 06 10:44:40.977060 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:40.977035    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:44:41.075380 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.074772    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpqzd\" (UniqueName: \"kubernetes.io/projected/5841eacf-26da-4a66-891a-d96f094f7e97-kube-api-access-cpqzd\") pod \"5841eacf-26da-4a66-891a-d96f094f7e97\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") "
Sep 06 10:44:41.075380 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.074890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-dshm\") pod \"5841eacf-26da-4a66-891a-d96f094f7e97\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") "
Sep 06 10:44:41.075380 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.074986    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-model-cache\") pod \"5841eacf-26da-4a66-891a-d96f094f7e97\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") "
Sep 06 10:44:41.075380 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.075105    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5841eacf-26da-4a66-891a-d96f094f7e97-tls-certs\") pod \"5841eacf-26da-4a66-891a-d96f094f7e97\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") "
Sep 06 10:44:41.075380 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.075143    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-tmp-dir\") pod \"5841eacf-26da-4a66-891a-d96f094f7e97\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") "
Sep 06 10:44:41.075380 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.075178    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-home\") pod \"5841eacf-26da-4a66-891a-d96f094f7e97\" (UID: \"5841eacf-26da-4a66-891a-d96f094f7e97\") "
Sep 06 10:44:41.076412 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.076036    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-home" (OuterVolumeSpecName: "home") pod "5841eacf-26da-4a66-891a-d96f094f7e97" (UID: "5841eacf-26da-4a66-891a-d96f094f7e97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:41.076412 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.076098    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-model-cache" (OuterVolumeSpecName: "model-cache") pod "5841eacf-26da-4a66-891a-d96f094f7e97" (UID: "5841eacf-26da-4a66-891a-d96f094f7e97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:41.076412 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.076282    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5841eacf-26da-4a66-891a-d96f094f7e97" (UID: "5841eacf-26da-4a66-891a-d96f094f7e97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:41.082291 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.082263    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5841eacf-26da-4a66-891a-d96f094f7e97-kube-api-access-cpqzd" (OuterVolumeSpecName: "kube-api-access-cpqzd") pod "5841eacf-26da-4a66-891a-d96f094f7e97" (UID: "5841eacf-26da-4a66-891a-d96f094f7e97"). InnerVolumeSpecName "kube-api-access-cpqzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:41.082562 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.082537    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5841eacf-26da-4a66-891a-d96f094f7e97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5841eacf-26da-4a66-891a-d96f094f7e97" (UID: "5841eacf-26da-4a66-891a-d96f094f7e97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:41.082708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.082676    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-dshm" (OuterVolumeSpecName: "dshm") pod "5841eacf-26da-4a66-891a-d96f094f7e97" (UID: "5841eacf-26da-4a66-891a-d96f094f7e97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:41.084513 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.084484    2564 generic.go:356] "Generic (PLEG): container finished" podID="5841eacf-26da-4a66-891a-d96f094f7e97" containerID="41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16" exitCode=0
Sep 06 10:44:41.084637 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.084551    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"
Sep 06 10:44:41.084637 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.084566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4" event={"ID":"5841eacf-26da-4a66-891a-d96f094f7e97","Type":"ContainerDied","Data":"41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16"}
Sep 06 10:44:41.084637 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.084613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4" event={"ID":"5841eacf-26da-4a66-891a-d96f094f7e97","Type":"ContainerDied","Data":"b1e6691daa6a345c66e6e1ded32412875e7cfda5d1b5e8c925db53c162f535bd"}
Sep 06 10:44:41.084637 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.084635    2564 scope.go:117] "RemoveContainer" containerID="41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16"
Sep 06 10:44:41.094345 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.094322    2564 scope.go:117] "RemoveContainer" containerID="41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16"
Sep 06 10:44:41.094592 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:44:41.094574    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16\": container with ID starting with 41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16 not found: ID does not exist" containerID="41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16"
Sep 06 10:44:41.094641 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.094597    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16"} err="failed to get container status \"41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16\": rpc error: code = NotFound desc = could not find container \"41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16\": container with ID starting with 41fac53683bf6520b5ebb8d08c226c81d879505b6c491b8608fc9d6765e68e16 not found: ID does not exist"
Sep 06 10:44:41.105968 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.105942    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"]
Sep 06 10:44:41.109543 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.109522    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/sub-test-model-682ea0-kserve-74799f8d67-6cdh4"]
Sep 06 10:44:41.176339 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.176312    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:41.176339 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.176336    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5841eacf-26da-4a66-891a-d96f094f7e97-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:41.176339 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.176344    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:41.176509 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.176352    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:41.176509 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.176360    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpqzd\" (UniqueName: \"kubernetes.io/projected/5841eacf-26da-4a66-891a-d96f094f7e97-kube-api-access-cpqzd\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:41.176509 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:41.176369    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5841eacf-26da-4a66-891a-d96f094f7e97-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:42.028970 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:42.028935    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5841eacf-26da-4a66-891a-d96f094f7e97" path="/var/lib/kubelet/pods/5841eacf-26da-4a66-891a-d96f094f7e97/volumes"
Sep 06 10:44:42.029378 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:42.029364    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef49ab32-70f1-4768-ad94-3a4aa5314bc7" path="/var/lib/kubelet/pods/ef49ab32-70f1-4768-ad94-3a4aa5314bc7/volumes"
Sep 06 10:44:44.497293 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497263    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"]
Sep 06 10:44:44.497843 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497823    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e2754f-136b-40d4-b99f-a8f25fd01ec9" containerName="payload-processing"
Sep 06 10:44:44.497933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497847    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e2754f-136b-40d4-b99f-a8f25fd01ec9" containerName="payload-processing"
Sep 06 10:44:44.497933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497862    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef49ab32-70f1-4768-ad94-3a4aa5314bc7" containerName="main"
Sep 06 10:44:44.497933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497870    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef49ab32-70f1-4768-ad94-3a4aa5314bc7" containerName="main"
Sep 06 10:44:44.497933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497894    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c8d031-cb9d-4c1b-98d9-8a1ea03df93b" containerName="payload-pre-processing"
Sep 06 10:44:44.497933 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497905    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c8d031-cb9d-4c1b-98d9-8a1ea03df93b" containerName="payload-pre-processing"
Sep 06 10:44:44.498168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497935    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44d6f55-dfe4-46a2-a7ba-15884230b38e" containerName="payload-pre-processing"
Sep 06 10:44:44.498168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497943    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44d6f55-dfe4-46a2-a7ba-15884230b38e" containerName="payload-pre-processing"
Sep 06 10:44:44.498168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497955    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5841eacf-26da-4a66-891a-d96f094f7e97" containerName="main"
Sep 06 10:44:44.498168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.497964    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5841eacf-26da-4a66-891a-d96f094f7e97" containerName="main"
Sep 06 10:44:44.498168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.498051    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44d6f55-dfe4-46a2-a7ba-15884230b38e" containerName="payload-pre-processing"
Sep 06 10:44:44.498168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.498065    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c8d031-cb9d-4c1b-98d9-8a1ea03df93b" containerName="payload-pre-processing"
Sep 06 10:44:44.498168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.498074    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5841eacf-26da-4a66-891a-d96f094f7e97" containerName="main"
Sep 06 10:44:44.498168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.498084    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e2754f-136b-40d4-b99f-a8f25fd01ec9" containerName="payload-processing"
Sep 06 10:44:44.498168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.498094    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef49ab32-70f1-4768-ad94-3a4aa5314bc7" containerName="main"
Sep 06 10:44:44.502605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.502585    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.506454 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.506433    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"rate-test-model-6751d7-kserve-self-signed-certs\""
Sep 06 10:44:44.506586 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.506503    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"kube-root-ca.crt\""
Sep 06 10:44:44.506586 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.506523    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"default-dockercfg-ktwrx\""
Sep 06 10:44:44.506707 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.506650    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6751d7\"/\"openshift-service-ca.crt\""
Sep 06 10:44:44.510787 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.510764    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"]
Sep 06 10:44:44.607969 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.607932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hr5w\" (UniqueName: \"kubernetes.io/projected/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-kube-api-access-7hr5w\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.608134 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.608027    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-home\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.608134 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.608078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tls-certs\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.608134 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.608105    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-model-cache\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.608343 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.608164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tmp-dir\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.608343 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.608190    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-dshm\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.709006 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.708963    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hr5w\" (UniqueName: \"kubernetes.io/projected/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-kube-api-access-7hr5w\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.709195 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.709026    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-home\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.709195 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.709050    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tls-certs\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.709195 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.709070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-model-cache\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.709195 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.709102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tmp-dir\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.709195 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.709122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-dshm\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.709537 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.709509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-home\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.709597 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.709516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-model-cache\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.709597 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.709557    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tmp-dir\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.711438 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.711416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-dshm\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.711720 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.711699    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tls-certs\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.718068 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.718044    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hr5w\" (UniqueName: \"kubernetes.io/projected/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-kube-api-access-7hr5w\") pod \"rate-test-model-6751d7-kserve-9c6c9f6dc-xblks\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") " pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:44.813248 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:44.813198    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:45.083564 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:45.083536    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"]
Sep 06 10:44:45.085545 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:44:45.085515    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c44e940_b42d_4ed4_a85b_55315f6fb7a6.slice/crio-93c7b3dc82480b9b7e6599d97bf79d5fd80ad4dd2181a3bed4c81d3a1a7e4b53 WatchSource:0}: Error finding container 93c7b3dc82480b9b7e6599d97bf79d5fd80ad4dd2181a3bed4c81d3a1a7e4b53: Status 404 returned error can't find the container with id 93c7b3dc82480b9b7e6599d97bf79d5fd80ad4dd2181a3bed4c81d3a1a7e4b53
Sep 06 10:44:45.108175 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:45.108152    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks" event={"ID":"7c44e940-b42d-4ed4-a85b-55315f6fb7a6","Type":"ContainerStarted","Data":"93c7b3dc82480b9b7e6599d97bf79d5fd80ad4dd2181a3bed4c81d3a1a7e4b53"}
Sep 06 10:44:46.114319 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:46.114282    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks" event={"ID":"7c44e940-b42d-4ed4-a85b-55315f6fb7a6","Type":"ContainerStarted","Data":"c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a"}
Sep 06 10:44:46.130500 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:46.130450    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks" podStartSLOduration=2.130434691 podStartE2EDuration="2.130434691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:44:46.129978427 +0000 UTC m=+1374.613478536" watchObservedRunningTime="2026-09-06 10:44:46.130434691 +0000 UTC m=+1374.613934799"
Sep 06 10:44:47.490916 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:47.490820    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"]
Sep 06 10:44:47.491400 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:47.491047    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg" podUID="c78180f7-be5e-48dd-90e3-fb497aa53b50" containerName="payload-pre-processing" containerID="cri-o://97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027" gracePeriod=30
Sep 06 10:44:47.753556 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:47.753492    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:44:47.840488 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:47.840457    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78180f7-be5e-48dd-90e3-fb497aa53b50-plugins-config-volume\") pod \"c78180f7-be5e-48dd-90e3-fb497aa53b50\" (UID: \"c78180f7-be5e-48dd-90e3-fb497aa53b50\") "
Sep 06 10:44:47.840644 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:47.840555    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqxqz\" (UniqueName: \"kubernetes.io/projected/c78180f7-be5e-48dd-90e3-fb497aa53b50-kube-api-access-pqxqz\") pod \"c78180f7-be5e-48dd-90e3-fb497aa53b50\" (UID: \"c78180f7-be5e-48dd-90e3-fb497aa53b50\") "
Sep 06 10:44:47.840818 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:47.840796    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78180f7-be5e-48dd-90e3-fb497aa53b50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c78180f7-be5e-48dd-90e3-fb497aa53b50" (UID: "c78180f7-be5e-48dd-90e3-fb497aa53b50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:47.842602 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:47.842573    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78180f7-be5e-48dd-90e3-fb497aa53b50-kube-api-access-pqxqz" (OuterVolumeSpecName: "kube-api-access-pqxqz") pod "c78180f7-be5e-48dd-90e3-fb497aa53b50" (UID: "c78180f7-be5e-48dd-90e3-fb497aa53b50"). InnerVolumeSpecName "kube-api-access-pqxqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:47.941370 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:47.941348    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqxqz\" (UniqueName: \"kubernetes.io/projected/c78180f7-be5e-48dd-90e3-fb497aa53b50-kube-api-access-pqxqz\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:47.941370 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:47.941368    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78180f7-be5e-48dd-90e3-fb497aa53b50-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:48.131260 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:48.131231    2564 generic.go:356] "Generic (PLEG): container finished" podID="c78180f7-be5e-48dd-90e3-fb497aa53b50" containerID="97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027" exitCode=0
Sep 06 10:44:48.131402 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:48.131270    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"
Sep 06 10:44:48.131402 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:48.131285    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg" event={"ID":"c78180f7-be5e-48dd-90e3-fb497aa53b50","Type":"ContainerDied","Data":"97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027"}
Sep 06 10:44:48.131402 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:48.131324    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg" event={"ID":"c78180f7-be5e-48dd-90e3-fb497aa53b50","Type":"ContainerDied","Data":"ad4e5b31e6774595d9127e3ad5cc8b800b3dc5cf0deea6e10bfc36a62a83580d"}
Sep 06 10:44:48.131402 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:48.131346    2564 scope.go:117] "RemoveContainer" containerID="97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027"
Sep 06 10:44:48.139919 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:48.139896    2564 scope.go:117] "RemoveContainer" containerID="97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027"
Sep 06 10:44:48.140186 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:44:48.140166    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027\": container with ID starting with 97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027 not found: ID does not exist" containerID="97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027"
Sep 06 10:44:48.140332 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:48.140194    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027"} err="failed to get container status \"97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027\": rpc error: code = NotFound desc = could not find container \"97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027\": container with ID starting with 97c44161da424e32cdff734fcda9800a84754398bfa5850ebbfc482e48239027 not found: ID does not exist"
Sep 06 10:44:48.148423 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:48.148399    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"]
Sep 06 10:44:48.152075 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:48.152055    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-27a2b9-5964dbcdf8-qmgxg"]
Sep 06 10:44:50.028874 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:50.028838    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c78180f7-be5e-48dd-90e3-fb497aa53b50" path="/var/lib/kubelet/pods/c78180f7-be5e-48dd-90e3-fb497aa53b50/volumes"
Sep 06 10:44:53.873202 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:53.873168    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"]
Sep 06 10:44:53.873671 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:53.873439    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy" containerID="cri-o://7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46" gracePeriod=30
Sep 06 10:44:54.814115 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:54.814073    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:54.814339 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:54.814136    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:54.827318 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:54.827291    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:54.841455 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:54.841424    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 06 10:44:54.841588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:54.841494    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 10:44:55.180052 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:55.179960    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:44:59.127227 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.127196    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:44:59.190357 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.190268    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerID="7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46" exitCode=0
Sep 06 10:44:59.190357 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.190342    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"
Sep 06 10:44:59.190526 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.190362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" event={"ID":"4a86f003-7ddc-4a8e-889d-bdcd3ed11239","Type":"ContainerDied","Data":"7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46"}
Sep 06 10:44:59.190526 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.190447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p" event={"ID":"4a86f003-7ddc-4a8e-889d-bdcd3ed11239","Type":"ContainerDied","Data":"545f6567980db746cc117bc65e9895807a0bd999fdad5430b97ad44d1311e1a7"}
Sep 06 10:44:59.190526 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.190486    2564 scope.go:117] "RemoveContainer" containerID="7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46"
Sep 06 10:44:59.199591 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.199572    2564 scope.go:117] "RemoveContainer" containerID="42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3"
Sep 06 10:44:59.207561 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.207541    2564 scope.go:117] "RemoveContainer" containerID="7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46"
Sep 06 10:44:59.207871 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:44:59.207849    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46\": container with ID starting with 7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46 not found: ID does not exist" containerID="7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46"
Sep 06 10:44:59.207924 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.207881    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46"} err="failed to get container status \"7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46\": rpc error: code = NotFound desc = could not find container \"7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46\": container with ID starting with 7acc82aa0b8381cd5277d92a60501a8ab98eb922ba326ae4bbe138c367a1ce46 not found: ID does not exist"
Sep 06 10:44:59.207924 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.207900    2564 scope.go:117] "RemoveContainer" containerID="42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3"
Sep 06 10:44:59.208150 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:44:59.208133    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3\": container with ID starting with 42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3 not found: ID does not exist" containerID="42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3"
Sep 06 10:44:59.208198 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.208156    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3"} err="failed to get container status \"42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3\": rpc error: code = NotFound desc = could not find container \"42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3\": container with ID starting with 42e7d4391331e43cfdf788b29976841a3c41fd3ca9c77238cbedfa09b45ed0d3 not found: ID does not exist"
Sep 06 10:44:59.243259 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243203    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-podinfo\") pod \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") "
Sep 06 10:44:59.243259 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243263    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istiod-ca-cert\") pod \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") "
Sep 06 10:44:59.243491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243298    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-data\") pod \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") "
Sep 06 10:44:59.243491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243338    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-token\") pod \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") "
Sep 06 10:44:59.243491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243360    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-envoy\") pod \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") "
Sep 06 10:44:59.243491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243420    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxhjl\" (UniqueName: \"kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-kube-api-access-vxhjl\") pod \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") "
Sep 06 10:44:59.243491 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243448    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-socket\") pod \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") "
Sep 06 10:44:59.243742 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243543    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-credential-socket\") pod \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") "
Sep 06 10:44:59.243742 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243584    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-certs\") pod \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\" (UID: \"4a86f003-7ddc-4a8e-889d-bdcd3ed11239\") "
Sep 06 10:44:59.243742 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243633    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a86f003-7ddc-4a8e-889d-bdcd3ed11239" (UID: "4a86f003-7ddc-4a8e-889d-bdcd3ed11239"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:59.243742 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243721    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a86f003-7ddc-4a8e-889d-bdcd3ed11239" (UID: "4a86f003-7ddc-4a8e-889d-bdcd3ed11239"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:59.243990 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243823    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a86f003-7ddc-4a8e-889d-bdcd3ed11239" (UID: "4a86f003-7ddc-4a8e-889d-bdcd3ed11239"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:59.243990 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243962    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a86f003-7ddc-4a8e-889d-bdcd3ed11239" (UID: "4a86f003-7ddc-4a8e-889d-bdcd3ed11239"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:59.244105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.243969    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istiod-ca-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:59.244105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.244016    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-data\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:59.244105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.244032    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-socket\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:59.244105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.244033    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a86f003-7ddc-4a8e-889d-bdcd3ed11239" (UID: "4a86f003-7ddc-4a8e-889d-bdcd3ed11239"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:59.245594 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.245564    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-kube-api-access-vxhjl" (OuterVolumeSpecName: "kube-api-access-vxhjl") pod "4a86f003-7ddc-4a8e-889d-bdcd3ed11239" (UID: "4a86f003-7ddc-4a8e-889d-bdcd3ed11239"). InnerVolumeSpecName "kube-api-access-vxhjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:59.245728 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.245660    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a86f003-7ddc-4a8e-889d-bdcd3ed11239" (UID: "4a86f003-7ddc-4a8e-889d-bdcd3ed11239"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 10:44:59.245957 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.245938    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a86f003-7ddc-4a8e-889d-bdcd3ed11239" (UID: "4a86f003-7ddc-4a8e-889d-bdcd3ed11239"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:59.246029 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.245951    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a86f003-7ddc-4a8e-889d-bdcd3ed11239" (UID: "4a86f003-7ddc-4a8e-889d-bdcd3ed11239"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:59.345328 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.345295    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-podinfo\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:59.345328 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.345325    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-token\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:59.345328 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.345335    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-istio-envoy\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:59.345562 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.345344    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxhjl\" (UniqueName: \"kubernetes.io/projected/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-kube-api-access-vxhjl\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:59.345562 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.345353    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-credential-socket\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:59.345562 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.345362    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a86f003-7ddc-4a8e-889d-bdcd3ed11239-workload-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:44:59.513297 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.513265    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"]
Sep 06 10:44:59.516920 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:44:59.516890    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-27a2b9-openshift-default-6fd54b878d-rv48p"]
Sep 06 10:45:00.036238 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:00.032768    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" path="/var/lib/kubelet/pods/4a86f003-7ddc-4a8e-889d-bdcd3ed11239/volumes"
Sep 06 10:45:04.929604 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:04.929567    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"]
Sep 06 10:45:04.930143 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:04.929846    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f" podUID="8ad9e14e-3d92-40be-a208-17f29bf06eb2" containerName="maas-api" containerID="cri-o://ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99" gracePeriod=30
Sep 06 10:45:05.178009 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.177982    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:45:05.223680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.223595    2564 generic.go:356] "Generic (PLEG): container finished" podID="8ad9e14e-3d92-40be-a208-17f29bf06eb2" containerID="ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99" exitCode=0
Sep 06 10:45:05.223680 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.223630    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f" event={"ID":"8ad9e14e-3d92-40be-a208-17f29bf06eb2","Type":"ContainerDied","Data":"ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99"}
Sep 06 10:45:05.223912 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.223688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f" event={"ID":"8ad9e14e-3d92-40be-a208-17f29bf06eb2","Type":"ContainerDied","Data":"9349ea35843f25fc7546fde54aeb9c73752a164b8da00cfa563f98927f6c910f"}
Sep 06 10:45:05.223912 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.223644    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"
Sep 06 10:45:05.223912 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.223713    2564 scope.go:117] "RemoveContainer" containerID="ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99"
Sep 06 10:45:05.232516 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.232494    2564 scope.go:117] "RemoveContainer" containerID="ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99"
Sep 06 10:45:05.232785 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:45:05.232766    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99\": container with ID starting with ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99 not found: ID does not exist" containerID="ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99"
Sep 06 10:45:05.232829 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.232795    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99"} err="failed to get container status \"ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99\": rpc error: code = NotFound desc = could not find container \"ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99\": container with ID starting with ba2e7f37a031b8bd639cc11378673766e67182825a7d88c98942ac86b5c5dd99 not found: ID does not exist"
Sep 06 10:45:05.300509 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.300482    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad9e14e-3d92-40be-a208-17f29bf06eb2-openshift-service-ca\") pod \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") "
Sep 06 10:45:05.300678 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.300587    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls\") pod \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") "
Sep 06 10:45:05.300678 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.300619    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g598q\" (UniqueName: \"kubernetes.io/projected/8ad9e14e-3d92-40be-a208-17f29bf06eb2-kube-api-access-g598q\") pod \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") "
Sep 06 10:45:05.300678 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.300645    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-metrics-certs\") pod \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\" (UID: \"8ad9e14e-3d92-40be-a208-17f29bf06eb2\") "
Sep 06 10:45:05.300855 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.300829    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad9e14e-3d92-40be-a208-17f29bf06eb2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ad9e14e-3d92-40be-a208-17f29bf06eb2" (UID: "8ad9e14e-3d92-40be-a208-17f29bf06eb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:45:05.302761 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.302727    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8ad9e14e-3d92-40be-a208-17f29bf06eb2" (UID: "8ad9e14e-3d92-40be-a208-17f29bf06eb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:45:05.302893 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.302784    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ad9e14e-3d92-40be-a208-17f29bf06eb2" (UID: "8ad9e14e-3d92-40be-a208-17f29bf06eb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:45:05.302893 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.302815    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad9e14e-3d92-40be-a208-17f29bf06eb2-kube-api-access-g598q" (OuterVolumeSpecName: "kube-api-access-g598q") pod "8ad9e14e-3d92-40be-a208-17f29bf06eb2" (UID: "8ad9e14e-3d92-40be-a208-17f29bf06eb2"). InnerVolumeSpecName "kube-api-access-g598q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:45:05.401507 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.401476    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-maas-api-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:45:05.401507 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.401503    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g598q\" (UniqueName: \"kubernetes.io/projected/8ad9e14e-3d92-40be-a208-17f29bf06eb2-kube-api-access-g598q\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:45:05.401507 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.401514    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad9e14e-3d92-40be-a208-17f29bf06eb2-metrics-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:45:05.401742 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.401523    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad9e14e-3d92-40be-a208-17f29bf06eb2-openshift-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:45:05.545841 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.545806    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"]
Sep 06 10:45:05.547979 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:05.547954    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71e210-5b78889b64-26z6f"]
Sep 06 10:45:06.029042 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:45:06.029003    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ad9e14e-3d92-40be-a208-17f29bf06eb2" path="/var/lib/kubelet/pods/8ad9e14e-3d92-40be-a208-17f29bf06eb2/volumes"
Sep 06 10:46:14.843244 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:14.843195    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"]
Sep 06 10:46:21.883277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883244    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"]
Sep 06 10:46:21.883676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883621    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78180f7-be5e-48dd-90e3-fb497aa53b50" containerName="payload-pre-processing"
Sep 06 10:46:21.883676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883633    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78180f7-be5e-48dd-90e3-fb497aa53b50" containerName="payload-pre-processing"
Sep 06 10:46:21.883676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883643    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy"
Sep 06 10:46:21.883676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883648    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy"
Sep 06 10:46:21.883676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883658    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy"
Sep 06 10:46:21.883676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883664    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy"
Sep 06 10:46:21.883676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883672    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad9e14e-3d92-40be-a208-17f29bf06eb2" containerName="maas-api"
Sep 06 10:46:21.883676 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883677    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad9e14e-3d92-40be-a208-17f29bf06eb2" containerName="maas-api"
Sep 06 10:46:21.883937 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883738    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c78180f7-be5e-48dd-90e3-fb497aa53b50" containerName="payload-pre-processing"
Sep 06 10:46:21.883937 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883748    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy"
Sep 06 10:46:21.883937 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883756    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a86f003-7ddc-4a8e-889d-bdcd3ed11239" containerName="istio-proxy"
Sep 06 10:46:21.883937 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.883762    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ad9e14e-3d92-40be-a208-17f29bf06eb2" containerName="maas-api"
Sep 06 10:46:21.888253 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.888232    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:46:21.891958 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.891934    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7d2fed-dockercfg-6qn75\""
Sep 06 10:46:21.891958 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.891952    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7d2fed\""
Sep 06 10:46:21.901358 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.901331    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"]
Sep 06 10:46:21.927875 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.927834    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413bc16a-520e-4bff-9a33-8a4b06621991-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589\" (UID: \"413bc16a-520e-4bff-9a33-8a4b06621991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:46:21.928025 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:21.927996    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cth5v\" (UniqueName: \"kubernetes.io/projected/413bc16a-520e-4bff-9a33-8a4b06621991-kube-api-access-cth5v\") pod \"payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589\" (UID: \"413bc16a-520e-4bff-9a33-8a4b06621991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:46:22.029347 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:22.029316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cth5v\" (UniqueName: \"kubernetes.io/projected/413bc16a-520e-4bff-9a33-8a4b06621991-kube-api-access-cth5v\") pod \"payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589\" (UID: \"413bc16a-520e-4bff-9a33-8a4b06621991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:46:22.029514 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:22.029366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413bc16a-520e-4bff-9a33-8a4b06621991-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589\" (UID: \"413bc16a-520e-4bff-9a33-8a4b06621991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:46:22.030039 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:22.030019    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413bc16a-520e-4bff-9a33-8a4b06621991-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589\" (UID: \"413bc16a-520e-4bff-9a33-8a4b06621991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:46:22.037320 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:22.037297    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cth5v\" (UniqueName: \"kubernetes.io/projected/413bc16a-520e-4bff-9a33-8a4b06621991-kube-api-access-cth5v\") pod \"payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589\" (UID: \"413bc16a-520e-4bff-9a33-8a4b06621991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:46:22.199044 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:22.198947    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:46:22.470394 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:22.470353    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"]
Sep 06 10:46:22.473054 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:46:22.473027    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413bc16a_520e_4bff_9a33_8a4b06621991.slice/crio-3b2955b838df26c60f965046bbb8e522f171b27762fa53cf0ef6fe52cd6f95cb WatchSource:0}: Error finding container 3b2955b838df26c60f965046bbb8e522f171b27762fa53cf0ef6fe52cd6f95cb: Status 404 returned error can't find the container with id 3b2955b838df26c60f965046bbb8e522f171b27762fa53cf0ef6fe52cd6f95cb
Sep 06 10:46:22.605406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:22.605372    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589" event={"ID":"413bc16a-520e-4bff-9a33-8a4b06621991","Type":"ContainerStarted","Data":"4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e"}
Sep 06 10:46:22.605593 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:22.605448    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589" event={"ID":"413bc16a-520e-4bff-9a33-8a4b06621991","Type":"ContainerStarted","Data":"3b2955b838df26c60f965046bbb8e522f171b27762fa53cf0ef6fe52cd6f95cb"}
Sep 06 10:46:22.605593 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:22.605510    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:46:22.621184 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:22.621132    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589" podStartSLOduration=1.6211128110000002 podStartE2EDuration="1.621112811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:46:22.620536805 +0000 UTC m=+1471.104036950" watchObservedRunningTime="2026-09-06 10:46:22.621112811 +0000 UTC m=+1471.104612921"
Sep 06 10:46:29.863032 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:29.862986    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks" podUID="7c44e940-b42d-4ed4-a85b-55315f6fb7a6" containerName="main" containerID="cri-o://c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a" gracePeriod=45
Sep 06 10:46:30.108961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.108939    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:46:30.206648 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.206556    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-dshm\") pod \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") "
Sep 06 10:46:30.206648 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.206600    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hr5w\" (UniqueName: \"kubernetes.io/projected/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-kube-api-access-7hr5w\") pod \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") "
Sep 06 10:46:30.206648 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.206629    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-home\") pod \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") "
Sep 06 10:46:30.206936 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.206655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tls-certs\") pod \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") "
Sep 06 10:46:30.206936 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.206698    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-model-cache\") pod \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") "
Sep 06 10:46:30.206936 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.206856    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tmp-dir\") pod \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\" (UID: \"7c44e940-b42d-4ed4-a85b-55315f6fb7a6\") "
Sep 06 10:46:30.207089 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.206950    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-home" (OuterVolumeSpecName: "home") pod "7c44e940-b42d-4ed4-a85b-55315f6fb7a6" (UID: "7c44e940-b42d-4ed4-a85b-55315f6fb7a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:46:30.207089 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.206966    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c44e940-b42d-4ed4-a85b-55315f6fb7a6" (UID: "7c44e940-b42d-4ed4-a85b-55315f6fb7a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:46:30.207175 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.207113    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c44e940-b42d-4ed4-a85b-55315f6fb7a6" (UID: "7c44e940-b42d-4ed4-a85b-55315f6fb7a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:46:30.207338 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.207320    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:46:30.207398 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.207344    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:46:30.207398 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.207359    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:46:30.208768 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.208741    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-dshm" (OuterVolumeSpecName: "dshm") pod "7c44e940-b42d-4ed4-a85b-55315f6fb7a6" (UID: "7c44e940-b42d-4ed4-a85b-55315f6fb7a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:46:30.208896 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.208782    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c44e940-b42d-4ed4-a85b-55315f6fb7a6" (UID: "7c44e940-b42d-4ed4-a85b-55315f6fb7a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:46:30.208896 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.208857    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-kube-api-access-7hr5w" (OuterVolumeSpecName: "kube-api-access-7hr5w") pod "7c44e940-b42d-4ed4-a85b-55315f6fb7a6" (UID: "7c44e940-b42d-4ed4-a85b-55315f6fb7a6"). InnerVolumeSpecName "kube-api-access-7hr5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:46:30.308587 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.308551    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:46:30.308587 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.308581    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hr5w\" (UniqueName: \"kubernetes.io/projected/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-kube-api-access-7hr5w\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:46:30.308587 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.308594    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c44e940-b42d-4ed4-a85b-55315f6fb7a6-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:46:30.645429 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.645394    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c44e940-b42d-4ed4-a85b-55315f6fb7a6" containerID="c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a" exitCode=0
Sep 06 10:46:30.645622 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.645434    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"
Sep 06 10:46:30.645622 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.645479    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks" event={"ID":"7c44e940-b42d-4ed4-a85b-55315f6fb7a6","Type":"ContainerDied","Data":"c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a"}
Sep 06 10:46:30.645622 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.645526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks" event={"ID":"7c44e940-b42d-4ed4-a85b-55315f6fb7a6","Type":"ContainerDied","Data":"93c7b3dc82480b9b7e6599d97bf79d5fd80ad4dd2181a3bed4c81d3a1a7e4b53"}
Sep 06 10:46:30.645622 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.645550    2564 scope.go:117] "RemoveContainer" containerID="c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a"
Sep 06 10:46:30.654673 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.654658    2564 scope.go:117] "RemoveContainer" containerID="c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a"
Sep 06 10:46:30.654927 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:46:30.654908    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a\": container with ID starting with c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a not found: ID does not exist" containerID="c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a"
Sep 06 10:46:30.654989 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.654939    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a"} err="failed to get container status \"c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a\": rpc error: code = NotFound desc = could not find container \"c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a\": container with ID starting with c13c8fbb094db6597dde5ee51500d8a97018cdc2081fb94d7dad8b5f8f6d7b7a not found: ID does not exist"
Sep 06 10:46:30.667000 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.666976    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"]
Sep 06 10:46:30.670372 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:30.670353    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6751d7/rate-test-model-6751d7-kserve-9c6c9f6dc-xblks"]
Sep 06 10:46:32.029115 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:32.029073    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c44e940-b42d-4ed4-a85b-55315f6fb7a6" path="/var/lib/kubelet/pods/7c44e940-b42d-4ed4-a85b-55315f6fb7a6/volumes"
Sep 06 10:46:33.612396 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:33.612368    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:46:50.371101 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.371060    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"]
Sep 06 10:46:50.371710 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.371692    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c44e940-b42d-4ed4-a85b-55315f6fb7a6" containerName="main"
Sep 06 10:46:50.371763 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.371714    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c44e940-b42d-4ed4-a85b-55315f6fb7a6" containerName="main"
Sep 06 10:46:50.371845 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.371833    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c44e940-b42d-4ed4-a85b-55315f6fb7a6" containerName="main"
Sep 06 10:46:50.383109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.383081    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.383862 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.383836    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"]
Sep 06 10:46:50.385881 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.385859    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5be549-serving-cert\""
Sep 06 10:46:50.401194 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.401166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf45f85-c474-404e-ae1e-ad2892a68bbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.401309 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.401262    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2s2\" (UniqueName: \"kubernetes.io/projected/faf45f85-c474-404e-ae1e-ad2892a68bbb-kube-api-access-nv2s2\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.401401 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.401385    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.401439 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.401421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-metrics-certs\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.502873 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.502838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.503040 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.502887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-metrics-certs\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.503040 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.502933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf45f85-c474-404e-ae1e-ad2892a68bbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.503040 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.502980    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2s2\" (UniqueName: \"kubernetes.io/projected/faf45f85-c474-404e-ae1e-ad2892a68bbb-kube-api-access-nv2s2\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.503040 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:46:50.502997    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-serving-cert: secret "maas-api-e2e-ipp-b-5be549-serving-cert" not found
Sep 06 10:46:50.503326 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:46:50.503093    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-maas-api-tls podName:faf45f85-c474-404e-ae1e-ad2892a68bbb nodeName:}" failed. No retries permitted until 2026-09-06 10:46:51.003071349 +0000 UTC m=+1499.486571435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-maas-api-tls") pod "maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b" (UID: "faf45f85-c474-404e-ae1e-ad2892a68bbb") : secret "maas-api-e2e-ipp-b-5be549-serving-cert" not found
Sep 06 10:46:50.503625 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.503605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf45f85-c474-404e-ae1e-ad2892a68bbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.505288 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.505263    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-metrics-certs\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:50.511689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:50.511665    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2s2\" (UniqueName: \"kubernetes.io/projected/faf45f85-c474-404e-ae1e-ad2892a68bbb-kube-api-access-nv2s2\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:51.008284 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.008244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:51.010659 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.010630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:51.028970 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.028941    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"]
Sep 06 10:46:51.032517 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.032500    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:46:51.035239 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.035206    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5be549\""
Sep 06 10:46:51.035376 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.035358    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5be549-dockercfg-k6l66\""
Sep 06 10:46:51.043569 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.043548    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"]
Sep 06 10:46:51.109013 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.108983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496d9abc-46c4-417e-8d6a-1dc064a32531-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74\" (UID: \"496d9abc-46c4-417e-8d6a-1dc064a32531\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:46:51.109175 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.109033    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2rf2\" (UniqueName: \"kubernetes.io/projected/496d9abc-46c4-417e-8d6a-1dc064a32531-kube-api-access-j2rf2\") pod \"payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74\" (UID: \"496d9abc-46c4-417e-8d6a-1dc064a32531\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:46:51.210082 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.210052    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496d9abc-46c4-417e-8d6a-1dc064a32531-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74\" (UID: \"496d9abc-46c4-417e-8d6a-1dc064a32531\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:46:51.210244 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.210099    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2rf2\" (UniqueName: \"kubernetes.io/projected/496d9abc-46c4-417e-8d6a-1dc064a32531-kube-api-access-j2rf2\") pod \"payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74\" (UID: \"496d9abc-46c4-417e-8d6a-1dc064a32531\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:46:51.210821 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.210800    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496d9abc-46c4-417e-8d6a-1dc064a32531-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74\" (UID: \"496d9abc-46c4-417e-8d6a-1dc064a32531\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:46:51.219157 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.219132    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2rf2\" (UniqueName: \"kubernetes.io/projected/496d9abc-46c4-417e-8d6a-1dc064a32531-kube-api-access-j2rf2\") pod \"payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74\" (UID: \"496d9abc-46c4-417e-8d6a-1dc064a32531\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:46:51.295778 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.295753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:51.343889 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.343844    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:46:51.569093 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.569070    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"]
Sep 06 10:46:51.570954 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:46:51.570925    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf45f85_c474_404e_ae1e_ad2892a68bbb.slice/crio-e8ab8159a4bc0e24593c4b397b50cdda0a191559e56a853dca6d60e974e972f2 WatchSource:0}: Error finding container e8ab8159a4bc0e24593c4b397b50cdda0a191559e56a853dca6d60e974e972f2: Status 404 returned error can't find the container with id e8ab8159a4bc0e24593c4b397b50cdda0a191559e56a853dca6d60e974e972f2
Sep 06 10:46:51.572455 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.572428    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:46:51.622768 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.622745    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"]
Sep 06 10:46:51.624899 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:46:51.624856    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496d9abc_46c4_417e_8d6a_1dc064a32531.slice/crio-7f147e39992f0ab9e1b9e4f8df8fd5d13830b9fa56cf6cbfe00f84a64e037e14 WatchSource:0}: Error finding container 7f147e39992f0ab9e1b9e4f8df8fd5d13830b9fa56cf6cbfe00f84a64e037e14: Status 404 returned error can't find the container with id 7f147e39992f0ab9e1b9e4f8df8fd5d13830b9fa56cf6cbfe00f84a64e037e14
Sep 06 10:46:51.747290 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.747239    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74" event={"ID":"496d9abc-46c4-417e-8d6a-1dc064a32531","Type":"ContainerStarted","Data":"8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be"}
Sep 06 10:46:51.747484 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.747310    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74" event={"ID":"496d9abc-46c4-417e-8d6a-1dc064a32531","Type":"ContainerStarted","Data":"7f147e39992f0ab9e1b9e4f8df8fd5d13830b9fa56cf6cbfe00f84a64e037e14"}
Sep 06 10:46:51.747484 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.747346    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:46:51.748615 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.748591    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b" event={"ID":"faf45f85-c474-404e-ae1e-ad2892a68bbb","Type":"ContainerStarted","Data":"e8ab8159a4bc0e24593c4b397b50cdda0a191559e56a853dca6d60e974e972f2"}
Sep 06 10:46:51.764605 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:51.764555    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74" podStartSLOduration=0.76454115 podStartE2EDuration="764.54115ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:46:51.763142161 +0000 UTC m=+1500.246642285" watchObservedRunningTime="2026-09-06 10:46:51.76454115 +0000 UTC m=+1500.248041258"
Sep 06 10:46:52.054646 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:52.054617    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:46:52.060383 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:52.060362    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:46:52.754896 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:52.754857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b" event={"ID":"faf45f85-c474-404e-ae1e-ad2892a68bbb","Type":"ContainerStarted","Data":"b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30"}
Sep 06 10:46:52.772464 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:52.772420    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b" podStartSLOduration=2.488048096 podStartE2EDuration="2.772405712s" totalImagesPullingTime="284.357616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:46:52.771492953 +0000 UTC m=+1501.254993073" watchObservedRunningTime="2026-09-06 10:46:52.772405712 +0000 UTC m=+1501.255905820"
Sep 06 10:46:53.759859 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:53.759819    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:46:59.769681 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:46:59.769651    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:47:02.757619 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:02.757589    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:47:19.499601 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:19.499513    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74_496d9abc-46c4-417e-8d6a-1dc064a32531/payload-processing/0.log"
Sep 06 10:47:32.992061 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:32.992032    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"]
Sep 06 10:47:32.995873 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:32.995850    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.001475 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.001450    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7d2fed\"/\"kube-root-ca.crt\""
Sep 06 10:47:33.001623 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.001512    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7d2fed\"/\"default-dockercfg-5hxlh\""
Sep 06 10:47:33.001704 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.001456    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7d2fed\"/\"ipp-route-7d2fed-kserve-self-signed-certs\""
Sep 06 10:47:33.002035 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.002016    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7d2fed\"/\"openshift-service-ca.crt\""
Sep 06 10:47:33.004961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.004940    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"]
Sep 06 10:47:33.085967 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.085940    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xnn\" (UniqueName: \"kubernetes.io/projected/c9584210-621b-498f-9467-c6555392a6d4-kube-api-access-n5xnn\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.086102 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.085997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9584210-621b-498f-9467-c6555392a6d4-tls-certs\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.086239 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.086095    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-dshm\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.086239 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.086132    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-model-cache\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.086369 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.086277    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-tmp-dir\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.086369 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.086346    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-home\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.186830 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.186796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9584210-621b-498f-9467-c6555392a6d4-tls-certs\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.186830 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.186834    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-dshm\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.187097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.186849    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-model-cache\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.187097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.186870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-tmp-dir\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.187097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.186895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-home\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.187097 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.186944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xnn\" (UniqueName: \"kubernetes.io/projected/c9584210-621b-498f-9467-c6555392a6d4-kube-api-access-n5xnn\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.187366 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.187347    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-tmp-dir\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.187431 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.187404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-model-cache\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.187474 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.187440    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-home\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.189133 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.189114    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-dshm\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.189472 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.189450    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9584210-621b-498f-9467-c6555392a6d4-tls-certs\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.195317 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.195295    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xnn\" (UniqueName: \"kubernetes.io/projected/c9584210-621b-498f-9467-c6555392a6d4-kube-api-access-n5xnn\") pod \"ipp-route-7d2fed-kserve-5db4f6bff9-phccd\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") " pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.309457 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.309428    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:33.580897 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.580869    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"]
Sep 06 10:47:33.582642 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:47:33.582612    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9584210_621b_498f_9467_c6555392a6d4.slice/crio-dcff896b6bf15333286fba3ab4246fd8a1d2bedbded995fc6d01fda35231cacf WatchSource:0}: Error finding container dcff896b6bf15333286fba3ab4246fd8a1d2bedbded995fc6d01fda35231cacf: Status 404 returned error can't find the container with id dcff896b6bf15333286fba3ab4246fd8a1d2bedbded995fc6d01fda35231cacf
Sep 06 10:47:33.972346 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.972247    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd" event={"ID":"c9584210-621b-498f-9467-c6555392a6d4","Type":"ContainerStarted","Data":"5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159"}
Sep 06 10:47:33.972346 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.972294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd" event={"ID":"c9584210-621b-498f-9467-c6555392a6d4","Type":"ContainerStarted","Data":"dcff896b6bf15333286fba3ab4246fd8a1d2bedbded995fc6d01fda35231cacf"}
Sep 06 10:47:33.989108 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:33.989054    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd" podStartSLOduration=1.989037915 podStartE2EDuration="1.989037915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:47:33.987715107 +0000 UTC m=+1542.471215214" watchObservedRunningTime="2026-09-06 10:47:33.989037915 +0000 UTC m=+1542.472538035"
Sep 06 10:47:43.310572 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:43.310530    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:43.310572 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:43.310579    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:43.323598 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:43.323573    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:44.036109 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:44.036070    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:47:59.364871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:59.364835    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589_413bc16a-520e-4bff-9a33-8a4b06621991/payload-processing/0.log"
Sep 06 10:47:59.611421 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:47:59.611394    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74_496d9abc-46c4-417e-8d6a-1dc064a32531/payload-processing/0.log"
Sep 06 10:49:01.361606 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:01.361568    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"]
Sep 06 10:49:01.364800 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:01.361876    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589" podUID="413bc16a-520e-4bff-9a33-8a4b06621991" containerName="payload-processing" containerID="cri-o://4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e" gracePeriod=30
Sep 06 10:49:01.602128 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:01.602102    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:49:01.728496 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:01.728407    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cth5v\" (UniqueName: \"kubernetes.io/projected/413bc16a-520e-4bff-9a33-8a4b06621991-kube-api-access-cth5v\") pod \"413bc16a-520e-4bff-9a33-8a4b06621991\" (UID: \"413bc16a-520e-4bff-9a33-8a4b06621991\") "
Sep 06 10:49:01.728496 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:01.728447    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413bc16a-520e-4bff-9a33-8a4b06621991-plugins-config-volume\") pod \"413bc16a-520e-4bff-9a33-8a4b06621991\" (UID: \"413bc16a-520e-4bff-9a33-8a4b06621991\") "
Sep 06 10:49:01.728825 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:01.728794    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413bc16a-520e-4bff-9a33-8a4b06621991-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "413bc16a-520e-4bff-9a33-8a4b06621991" (UID: "413bc16a-520e-4bff-9a33-8a4b06621991"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:49:01.730539 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:01.730505    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413bc16a-520e-4bff-9a33-8a4b06621991-kube-api-access-cth5v" (OuterVolumeSpecName: "kube-api-access-cth5v") pod "413bc16a-520e-4bff-9a33-8a4b06621991" (UID: "413bc16a-520e-4bff-9a33-8a4b06621991"). InnerVolumeSpecName "kube-api-access-cth5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:49:01.829129 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:01.829092    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cth5v\" (UniqueName: \"kubernetes.io/projected/413bc16a-520e-4bff-9a33-8a4b06621991-kube-api-access-cth5v\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:49:01.829129 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:01.829121    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413bc16a-520e-4bff-9a33-8a4b06621991-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:49:02.427095 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:02.426996    2564 generic.go:356] "Generic (PLEG): container finished" podID="413bc16a-520e-4bff-9a33-8a4b06621991" containerID="4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e" exitCode=0
Sep 06 10:49:02.427095 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:02.427059    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"
Sep 06 10:49:02.427095 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:02.427075    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589" event={"ID":"413bc16a-520e-4bff-9a33-8a4b06621991","Type":"ContainerDied","Data":"4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e"}
Sep 06 10:49:02.427666 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:02.427121    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589" event={"ID":"413bc16a-520e-4bff-9a33-8a4b06621991","Type":"ContainerDied","Data":"3b2955b838df26c60f965046bbb8e522f171b27762fa53cf0ef6fe52cd6f95cb"}
Sep 06 10:49:02.427666 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:02.427142    2564 scope.go:117] "RemoveContainer" containerID="4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e"
Sep 06 10:49:02.436274 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:02.436257    2564 scope.go:117] "RemoveContainer" containerID="4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e"
Sep 06 10:49:02.436552 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:49:02.436532    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e\": container with ID starting with 4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e not found: ID does not exist" containerID="4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e"
Sep 06 10:49:02.436599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:02.436560    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e"} err="failed to get container status \"4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e\": rpc error: code = NotFound desc = could not find container \"4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e\": container with ID starting with 4d83f6d29cb1e714bccf8f6722a13e2896630378ff5f6a6c94dd1e06ce17af3e not found: ID does not exist"
Sep 06 10:49:02.444527 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:02.444504    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"]
Sep 06 10:49:02.449119 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:02.449095    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d2fed-6888fcd869-fz589"]
Sep 06 10:49:04.029035 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:04.029003    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="413bc16a-520e-4bff-9a33-8a4b06621991" path="/var/lib/kubelet/pods/413bc16a-520e-4bff-9a33-8a4b06621991/volumes"
Sep 06 10:49:06.906686 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:06.906651    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"]
Sep 06 10:49:21.926568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:21.926519    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd" podUID="c9584210-621b-498f-9467-c6555392a6d4" containerName="main" containerID="cri-o://5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159" gracePeriod=45
Sep 06 10:49:22.175222 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.175186    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:49:22.323233 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323182    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5xnn\" (UniqueName: \"kubernetes.io/projected/c9584210-621b-498f-9467-c6555392a6d4-kube-api-access-n5xnn\") pod \"c9584210-621b-498f-9467-c6555392a6d4\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") "
Sep 06 10:49:22.323434 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323244    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-home\") pod \"c9584210-621b-498f-9467-c6555392a6d4\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") "
Sep 06 10:49:22.323434 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323297    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-tmp-dir\") pod \"c9584210-621b-498f-9467-c6555392a6d4\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") "
Sep 06 10:49:22.323434 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323347    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-dshm\") pod \"c9584210-621b-498f-9467-c6555392a6d4\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") "
Sep 06 10:49:22.323434 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323367    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9584210-621b-498f-9467-c6555392a6d4-tls-certs\") pod \"c9584210-621b-498f-9467-c6555392a6d4\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") "
Sep 06 10:49:22.323434 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323406    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-model-cache\") pod \"c9584210-621b-498f-9467-c6555392a6d4\" (UID: \"c9584210-621b-498f-9467-c6555392a6d4\") "
Sep 06 10:49:22.323701 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323547    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-home" (OuterVolumeSpecName: "home") pod "c9584210-621b-498f-9467-c6555392a6d4" (UID: "c9584210-621b-498f-9467-c6555392a6d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:49:22.323701 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323561    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9584210-621b-498f-9467-c6555392a6d4" (UID: "c9584210-621b-498f-9467-c6555392a6d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:49:22.323820 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323787    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9584210-621b-498f-9467-c6555392a6d4" (UID: "c9584210-621b-498f-9467-c6555392a6d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:49:22.323820 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323796    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:49:22.323923 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.323831    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:49:22.325540 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.325512    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-dshm" (OuterVolumeSpecName: "dshm") pod "c9584210-621b-498f-9467-c6555392a6d4" (UID: "c9584210-621b-498f-9467-c6555392a6d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:49:22.325677 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.325567    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9584210-621b-498f-9467-c6555392a6d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9584210-621b-498f-9467-c6555392a6d4" (UID: "c9584210-621b-498f-9467-c6555392a6d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:49:22.325677 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.325582    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9584210-621b-498f-9467-c6555392a6d4-kube-api-access-n5xnn" (OuterVolumeSpecName: "kube-api-access-n5xnn") pod "c9584210-621b-498f-9467-c6555392a6d4" (UID: "c9584210-621b-498f-9467-c6555392a6d4"). InnerVolumeSpecName "kube-api-access-n5xnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:49:22.424284 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.424251    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:49:22.424284 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.424278    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9584210-621b-498f-9467-c6555392a6d4-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:49:22.424284 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.424293    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9584210-621b-498f-9467-c6555392a6d4-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:49:22.424524 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.424305    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xnn\" (UniqueName: \"kubernetes.io/projected/c9584210-621b-498f-9467-c6555392a6d4-kube-api-access-n5xnn\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:49:22.525904 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.525868    2564 generic.go:356] "Generic (PLEG): container finished" podID="c9584210-621b-498f-9467-c6555392a6d4" containerID="5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159" exitCode=0
Sep 06 10:49:22.526105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.525909    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"
Sep 06 10:49:22.526105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.525953    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd" event={"ID":"c9584210-621b-498f-9467-c6555392a6d4","Type":"ContainerDied","Data":"5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159"}
Sep 06 10:49:22.526105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.526003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd" event={"ID":"c9584210-621b-498f-9467-c6555392a6d4","Type":"ContainerDied","Data":"dcff896b6bf15333286fba3ab4246fd8a1d2bedbded995fc6d01fda35231cacf"}
Sep 06 10:49:22.526105 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.526024    2564 scope.go:117] "RemoveContainer" containerID="5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159"
Sep 06 10:49:22.535267 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.535246    2564 scope.go:117] "RemoveContainer" containerID="5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159"
Sep 06 10:49:22.535574 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:49:22.535553    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159\": container with ID starting with 5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159 not found: ID does not exist" containerID="5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159"
Sep 06 10:49:22.535642 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.535581    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159"} err="failed to get container status \"5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159\": rpc error: code = NotFound desc = could not find container \"5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159\": container with ID starting with 5735c5140d7968c72361b1f491bec15d94815d0fc9dbc8ed2bc87a00e21ee159 not found: ID does not exist"
Sep 06 10:49:22.547355 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.547330    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"]
Sep 06 10:49:22.549830 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:22.549811    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7d2fed/ipp-route-7d2fed-kserve-5db4f6bff9-phccd"]
Sep 06 10:49:24.029113 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:49:24.029084    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9584210-621b-498f-9467-c6555392a6d4" path="/var/lib/kubelet/pods/c9584210-621b-498f-9467-c6555392a6d4/volumes"
Sep 06 10:50:00.797505 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:00.797472    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"]
Sep 06 10:50:00.798064 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:00.797734    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b" podUID="faf45f85-c474-404e-ae1e-ad2892a68bbb" containerName="maas-api" containerID="cri-o://b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30" gracePeriod=30
Sep 06 10:50:00.852048 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:00.851426    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"]
Sep 06 10:50:00.852048 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:00.851700    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74" podUID="496d9abc-46c4-417e-8d6a-1dc064a32531" containerName="payload-processing" containerID="cri-o://8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be" gracePeriod=30
Sep 06 10:50:01.113179 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.113158    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:50:01.116420 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.116396    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:50:01.272232 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.272187    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf45f85-c474-404e-ae1e-ad2892a68bbb-openshift-service-ca\") pod \"faf45f85-c474-404e-ae1e-ad2892a68bbb\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") "
Sep 06 10:50:01.272432 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.272246    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496d9abc-46c4-417e-8d6a-1dc064a32531-plugins-config-volume\") pod \"496d9abc-46c4-417e-8d6a-1dc064a32531\" (UID: \"496d9abc-46c4-417e-8d6a-1dc064a32531\") "
Sep 06 10:50:01.272432 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.272279    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-maas-api-tls\") pod \"faf45f85-c474-404e-ae1e-ad2892a68bbb\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") "
Sep 06 10:50:01.272432 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.272302    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2rf2\" (UniqueName: \"kubernetes.io/projected/496d9abc-46c4-417e-8d6a-1dc064a32531-kube-api-access-j2rf2\") pod \"496d9abc-46c4-417e-8d6a-1dc064a32531\" (UID: \"496d9abc-46c4-417e-8d6a-1dc064a32531\") "
Sep 06 10:50:01.272432 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.272369    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv2s2\" (UniqueName: \"kubernetes.io/projected/faf45f85-c474-404e-ae1e-ad2892a68bbb-kube-api-access-nv2s2\") pod \"faf45f85-c474-404e-ae1e-ad2892a68bbb\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") "
Sep 06 10:50:01.272432 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.272398    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-metrics-certs\") pod \"faf45f85-c474-404e-ae1e-ad2892a68bbb\" (UID: \"faf45f85-c474-404e-ae1e-ad2892a68bbb\") "
Sep 06 10:50:01.272729 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.272628    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf45f85-c474-404e-ae1e-ad2892a68bbb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "faf45f85-c474-404e-ae1e-ad2892a68bbb" (UID: "faf45f85-c474-404e-ae1e-ad2892a68bbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:50:01.272729 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.272642    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496d9abc-46c4-417e-8d6a-1dc064a32531-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "496d9abc-46c4-417e-8d6a-1dc064a32531" (UID: "496d9abc-46c4-417e-8d6a-1dc064a32531"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:50:01.272822 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.272744    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496d9abc-46c4-417e-8d6a-1dc064a32531-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:01.272822 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.272764    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf45f85-c474-404e-ae1e-ad2892a68bbb-openshift-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:01.274837 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.274802    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf45f85-c474-404e-ae1e-ad2892a68bbb-kube-api-access-nv2s2" (OuterVolumeSpecName: "kube-api-access-nv2s2") pod "faf45f85-c474-404e-ae1e-ad2892a68bbb" (UID: "faf45f85-c474-404e-ae1e-ad2892a68bbb"). InnerVolumeSpecName "kube-api-access-nv2s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:01.274837 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.274845    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496d9abc-46c4-417e-8d6a-1dc064a32531-kube-api-access-j2rf2" (OuterVolumeSpecName: "kube-api-access-j2rf2") pod "496d9abc-46c4-417e-8d6a-1dc064a32531" (UID: "496d9abc-46c4-417e-8d6a-1dc064a32531"). InnerVolumeSpecName "kube-api-access-j2rf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:01.275054 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.275023    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "faf45f85-c474-404e-ae1e-ad2892a68bbb" (UID: "faf45f85-c474-404e-ae1e-ad2892a68bbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:50:01.275119 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.275107    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "faf45f85-c474-404e-ae1e-ad2892a68bbb" (UID: "faf45f85-c474-404e-ae1e-ad2892a68bbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:50:01.374279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.374167    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-maas-api-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:01.374279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.374219    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2rf2\" (UniqueName: \"kubernetes.io/projected/496d9abc-46c4-417e-8d6a-1dc064a32531-kube-api-access-j2rf2\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:01.374279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.374232    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv2s2\" (UniqueName: \"kubernetes.io/projected/faf45f85-c474-404e-ae1e-ad2892a68bbb-kube-api-access-nv2s2\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:01.374279 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.374240    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf45f85-c474-404e-ae1e-ad2892a68bbb-metrics-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:01.724571 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.724467    2564 generic.go:356] "Generic (PLEG): container finished" podID="496d9abc-46c4-417e-8d6a-1dc064a32531" containerID="8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be" exitCode=0
Sep 06 10:50:01.724571 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.724511    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"
Sep 06 10:50:01.724795 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.724565    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74" event={"ID":"496d9abc-46c4-417e-8d6a-1dc064a32531","Type":"ContainerDied","Data":"8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be"}
Sep 06 10:50:01.724795 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.724621    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74" event={"ID":"496d9abc-46c4-417e-8d6a-1dc064a32531","Type":"ContainerDied","Data":"7f147e39992f0ab9e1b9e4f8df8fd5d13830b9fa56cf6cbfe00f84a64e037e14"}
Sep 06 10:50:01.724795 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.724653    2564 scope.go:117] "RemoveContainer" containerID="8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be"
Sep 06 10:50:01.726034 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.726008    2564 generic.go:356] "Generic (PLEG): container finished" podID="faf45f85-c474-404e-ae1e-ad2892a68bbb" containerID="b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30" exitCode=0
Sep 06 10:50:01.726120 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.726040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b" event={"ID":"faf45f85-c474-404e-ae1e-ad2892a68bbb","Type":"ContainerDied","Data":"b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30"}
Sep 06 10:50:01.726120 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.726057    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"
Sep 06 10:50:01.726120 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.726089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b" event={"ID":"faf45f85-c474-404e-ae1e-ad2892a68bbb","Type":"ContainerDied","Data":"e8ab8159a4bc0e24593c4b397b50cdda0a191559e56a853dca6d60e974e972f2"}
Sep 06 10:50:01.735370 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.735349    2564 scope.go:117] "RemoveContainer" containerID="8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be"
Sep 06 10:50:01.735702 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:50:01.735683    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be\": container with ID starting with 8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be not found: ID does not exist" containerID="8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be"
Sep 06 10:50:01.735793 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.735707    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be"} err="failed to get container status \"8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be\": rpc error: code = NotFound desc = could not find container \"8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be\": container with ID starting with 8f5da7b13aae609229bb90d3eef1cb55550a218642d113773265990d4065c2be not found: ID does not exist"
Sep 06 10:50:01.735793 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.735724    2564 scope.go:117] "RemoveContainer" containerID="b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30"
Sep 06 10:50:01.743714 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.743696    2564 scope.go:117] "RemoveContainer" containerID="b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30"
Sep 06 10:50:01.743950 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:50:01.743930    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30\": container with ID starting with b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30 not found: ID does not exist" containerID="b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30"
Sep 06 10:50:01.744000 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.743957    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30"} err="failed to get container status \"b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30\": rpc error: code = NotFound desc = could not find container \"b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30\": container with ID starting with b89204fcbb4ac196861ef7f124fa991a563d4b2a5a317fc108acc372a2175b30 not found: ID does not exist"
Sep 06 10:50:01.751663 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.751644    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"]
Sep 06 10:50:01.756117 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.756097    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5be549-5d7f69798c-ppr74"]
Sep 06 10:50:01.766614 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.766593    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"]
Sep 06 10:50:01.770687 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:01.770667    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5be549-5d58fcdb8d-g985b"]
Sep 06 10:50:02.029361 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:02.029282    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="496d9abc-46c4-417e-8d6a-1dc064a32531" path="/var/lib/kubelet/pods/496d9abc-46c4-417e-8d6a-1dc064a32531/volumes"
Sep 06 10:50:02.029734 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:02.029720    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faf45f85-c474-404e-ae1e-ad2892a68bbb" path="/var/lib/kubelet/pods/faf45f85-c474-404e-ae1e-ad2892a68bbb/volumes"
Sep 06 10:50:18.403954 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.403880    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"]
Sep 06 10:50:18.404407 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.404065    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx" podUID="2fac2a2c-bc1e-4954-bdc7-8c5954143d2a" containerName="maas-api" containerID="cri-o://fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321" gracePeriod=30
Sep 06 10:50:18.465442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.465408    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"]
Sep 06 10:50:18.465683 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.465623    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf" podUID="37ae96c9-a528-499e-98f5-a805cc2ceb6b" containerName="payload-processing" containerID="cri-o://4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb" gracePeriod=30
Sep 06 10:50:18.706950 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.706923    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:50:18.723199 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.723173    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xc4z\" (UniqueName: \"kubernetes.io/projected/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-kube-api-access-2xc4z\") pod \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") "
Sep 06 10:50:18.723356 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.723228    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-openshift-service-ca\") pod \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") "
Sep 06 10:50:18.723356 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.723276    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-metrics-certs\") pod \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") "
Sep 06 10:50:18.723356 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.723339    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-maas-api-tls\") pod \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\" (UID: \"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a\") "
Sep 06 10:50:18.725003 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.724658    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2fac2a2c-bc1e-4954-bdc7-8c5954143d2a" (UID: "2fac2a2c-bc1e-4954-bdc7-8c5954143d2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:50:18.727656 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.727616    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2fac2a2c-bc1e-4954-bdc7-8c5954143d2a" (UID: "2fac2a2c-bc1e-4954-bdc7-8c5954143d2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:50:18.727852 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.727829    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2fac2a2c-bc1e-4954-bdc7-8c5954143d2a" (UID: "2fac2a2c-bc1e-4954-bdc7-8c5954143d2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:50:18.727998 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.727925    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-kube-api-access-2xc4z" (OuterVolumeSpecName: "kube-api-access-2xc4z") pod "2fac2a2c-bc1e-4954-bdc7-8c5954143d2a" (UID: "2fac2a2c-bc1e-4954-bdc7-8c5954143d2a"). InnerVolumeSpecName "kube-api-access-2xc4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:18.733802 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.733781    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:50:18.809873 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.809844    2564 generic.go:356] "Generic (PLEG): container finished" podID="2fac2a2c-bc1e-4954-bdc7-8c5954143d2a" containerID="fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321" exitCode=0
Sep 06 10:50:18.810065 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.809889    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"
Sep 06 10:50:18.810065 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.809933    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx" event={"ID":"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a","Type":"ContainerDied","Data":"fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321"}
Sep 06 10:50:18.810065 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.809987    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx" event={"ID":"2fac2a2c-bc1e-4954-bdc7-8c5954143d2a","Type":"ContainerDied","Data":"ff89892e7aef2c78072659ba36840ba08d263da15230644b3af1df02498809a2"}
Sep 06 10:50:18.810065 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.810016    2564 scope.go:117] "RemoveContainer" containerID="fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321"
Sep 06 10:50:18.811459 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.811440    2564 generic.go:356] "Generic (PLEG): container finished" podID="37ae96c9-a528-499e-98f5-a805cc2ceb6b" containerID="4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb" exitCode=0
Sep 06 10:50:18.811573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.811470    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf" event={"ID":"37ae96c9-a528-499e-98f5-a805cc2ceb6b","Type":"ContainerDied","Data":"4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb"}
Sep 06 10:50:18.811573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.811529    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf" event={"ID":"37ae96c9-a528-499e-98f5-a805cc2ceb6b","Type":"ContainerDied","Data":"2528387ed87a2f324af0fb6634ef48f4157a9af1a71fc32f5ebe6a325448a5a9"}
Sep 06 10:50:18.811573 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.811480    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"
Sep 06 10:50:18.819546 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.819531    2564 scope.go:117] "RemoveContainer" containerID="fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321"
Sep 06 10:50:18.819810 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:50:18.819792    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321\": container with ID starting with fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321 not found: ID does not exist" containerID="fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321"
Sep 06 10:50:18.819877 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.819822    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321"} err="failed to get container status \"fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321\": rpc error: code = NotFound desc = could not find container \"fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321\": container with ID starting with fe141ca503255c27d483c99d22203672dea06de5d0de9e0446c03c673288a321 not found: ID does not exist"
Sep 06 10:50:18.819877 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.819844    2564 scope.go:117] "RemoveContainer" containerID="4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb"
Sep 06 10:50:18.824383 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.824362    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56x79\" (UniqueName: \"kubernetes.io/projected/37ae96c9-a528-499e-98f5-a805cc2ceb6b-kube-api-access-56x79\") pod \"37ae96c9-a528-499e-98f5-a805cc2ceb6b\" (UID: \"37ae96c9-a528-499e-98f5-a805cc2ceb6b\") "
Sep 06 10:50:18.824465 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.824451    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37ae96c9-a528-499e-98f5-a805cc2ceb6b-plugins-config-volume\") pod \"37ae96c9-a528-499e-98f5-a805cc2ceb6b\" (UID: \"37ae96c9-a528-499e-98f5-a805cc2ceb6b\") "
Sep 06 10:50:18.824758 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.824737    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ae96c9-a528-499e-98f5-a805cc2ceb6b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37ae96c9-a528-499e-98f5-a805cc2ceb6b" (UID: "37ae96c9-a528-499e-98f5-a805cc2ceb6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:50:18.824816 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.824781    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xc4z\" (UniqueName: \"kubernetes.io/projected/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-kube-api-access-2xc4z\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:18.824816 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.824800    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-openshift-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:18.824816 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.824814    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-metrics-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:18.825028 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.824827    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a-maas-api-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:18.826419 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.826398    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ae96c9-a528-499e-98f5-a805cc2ceb6b-kube-api-access-56x79" (OuterVolumeSpecName: "kube-api-access-56x79") pod "37ae96c9-a528-499e-98f5-a805cc2ceb6b" (UID: "37ae96c9-a528-499e-98f5-a805cc2ceb6b"). InnerVolumeSpecName "kube-api-access-56x79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:18.827553 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.827538    2564 scope.go:117] "RemoveContainer" containerID="4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb"
Sep 06 10:50:18.827804 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:50:18.827786    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb\": container with ID starting with 4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb not found: ID does not exist" containerID="4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb"
Sep 06 10:50:18.827864 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.827808    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb"} err="failed to get container status \"4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb\": rpc error: code = NotFound desc = could not find container \"4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb\": container with ID starting with 4f8f2d3f1d63dae87832842921ef1e1719417c70b2bea818f891d6bc606956bb not found: ID does not exist"
Sep 06 10:50:18.831648 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.831624    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"]
Sep 06 10:50:18.835960 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.835940    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6751d7-7746f6b7bb-4l4mx"]
Sep 06 10:50:18.925913 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.925880    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56x79\" (UniqueName: \"kubernetes.io/projected/37ae96c9-a528-499e-98f5-a805cc2ceb6b-kube-api-access-56x79\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:18.925913 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:18.925910    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37ae96c9-a528-499e-98f5-a805cc2ceb6b-plugins-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:19.132948 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:19.132921    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"]
Sep 06 10:50:19.136611 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:19.136582    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6751d7-877699687-dtfsf"]
Sep 06 10:50:20.029443 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:20.029410    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fac2a2c-bc1e-4954-bdc7-8c5954143d2a" path="/var/lib/kubelet/pods/2fac2a2c-bc1e-4954-bdc7-8c5954143d2a/volumes"
Sep 06 10:50:20.029801 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:20.029781    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ae96c9-a528-499e-98f5-a805cc2ceb6b" path="/var/lib/kubelet/pods/37ae96c9-a528-499e-98f5-a805cc2ceb6b/volumes"
Sep 06 10:50:36.053700 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.053663    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"]
Sep 06 10:50:36.054407 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.053892    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz" podUID="1aeff591-42ec-419b-b7cb-bd18d25e0415" containerName="maas-api" containerID="cri-o://4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580" gracePeriod=30
Sep 06 10:50:36.290813 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.290783    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:50:36.378916 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.378827    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6j2w\" (UniqueName: \"kubernetes.io/projected/1aeff591-42ec-419b-b7cb-bd18d25e0415-kube-api-access-s6j2w\") pod \"1aeff591-42ec-419b-b7cb-bd18d25e0415\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") "
Sep 06 10:50:36.378916 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.378887    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-metrics-certs\") pod \"1aeff591-42ec-419b-b7cb-bd18d25e0415\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") "
Sep 06 10:50:36.379111 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.378930    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-maas-api-tls\") pod \"1aeff591-42ec-419b-b7cb-bd18d25e0415\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") "
Sep 06 10:50:36.379111 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.378971    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeff591-42ec-419b-b7cb-bd18d25e0415-openshift-service-ca\") pod \"1aeff591-42ec-419b-b7cb-bd18d25e0415\" (UID: \"1aeff591-42ec-419b-b7cb-bd18d25e0415\") "
Sep 06 10:50:36.379424 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.379394    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aeff591-42ec-419b-b7cb-bd18d25e0415-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1aeff591-42ec-419b-b7cb-bd18d25e0415" (UID: "1aeff591-42ec-419b-b7cb-bd18d25e0415"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:50:36.381063 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.381039    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1aeff591-42ec-419b-b7cb-bd18d25e0415" (UID: "1aeff591-42ec-419b-b7cb-bd18d25e0415"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:50:36.381185 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.381067    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aeff591-42ec-419b-b7cb-bd18d25e0415-kube-api-access-s6j2w" (OuterVolumeSpecName: "kube-api-access-s6j2w") pod "1aeff591-42ec-419b-b7cb-bd18d25e0415" (UID: "1aeff591-42ec-419b-b7cb-bd18d25e0415"). InnerVolumeSpecName "kube-api-access-s6j2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:36.381185 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.381075    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1aeff591-42ec-419b-b7cb-bd18d25e0415" (UID: "1aeff591-42ec-419b-b7cb-bd18d25e0415"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:50:36.480652 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.480619    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-metrics-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:36.480652 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.480646    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aeff591-42ec-419b-b7cb-bd18d25e0415-maas-api-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:36.480652 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.480658    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeff591-42ec-419b-b7cb-bd18d25e0415-openshift-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:36.480871 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.480671    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6j2w\" (UniqueName: \"kubernetes.io/projected/1aeff591-42ec-419b-b7cb-bd18d25e0415-kube-api-access-s6j2w\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:50:36.898313 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.898277    2564 generic.go:356] "Generic (PLEG): container finished" podID="1aeff591-42ec-419b-b7cb-bd18d25e0415" containerID="4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580" exitCode=0
Sep 06 10:50:36.898500 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.898330    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"
Sep 06 10:50:36.898500 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.898365    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz" event={"ID":"1aeff591-42ec-419b-b7cb-bd18d25e0415","Type":"ContainerDied","Data":"4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580"}
Sep 06 10:50:36.898500 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.898422    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz" event={"ID":"1aeff591-42ec-419b-b7cb-bd18d25e0415","Type":"ContainerDied","Data":"d448a2c3fbd48fab8f16d257d8c6a3923869a637a114b45ec159e17af660af17"}
Sep 06 10:50:36.898500 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.898445    2564 scope.go:117] "RemoveContainer" containerID="4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580"
Sep 06 10:50:36.907784 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.907762    2564 scope.go:117] "RemoveContainer" containerID="4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580"
Sep 06 10:50:36.908033 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:50:36.908014    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580\": container with ID starting with 4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580 not found: ID does not exist" containerID="4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580"
Sep 06 10:50:36.908088 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.908042    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580"} err="failed to get container status \"4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580\": rpc error: code = NotFound desc = could not find container \"4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580\": container with ID starting with 4d5a0448bc7388fe7a9ec81368514f8b1dd48b4d4c8cd2c251ff654a6bd7e580 not found: ID does not exist"
Sep 06 10:50:36.920356 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.920332    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"]
Sep 06 10:50:36.924190 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:36.924168    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-682ea0-65d5545b55-hddkz"]
Sep 06 10:50:38.028748 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:50:38.028700    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aeff591-42ec-419b-b7cb-bd18d25e0415" path="/var/lib/kubelet/pods/1aeff591-42ec-419b-b7cb-bd18d25e0415/volumes"
Sep 06 10:51:52.084539 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:51:52.084512    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:51:52.092999 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:51:52.092968    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:55:11.587469 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587434    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"]
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587779    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faf45f85-c474-404e-ae1e-ad2892a68bbb" containerName="maas-api"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587790    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="faf45f85-c474-404e-ae1e-ad2892a68bbb" containerName="maas-api"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587805    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9584210-621b-498f-9467-c6555392a6d4" containerName="main"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587811    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9584210-621b-498f-9467-c6555392a6d4" containerName="main"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587818    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496d9abc-46c4-417e-8d6a-1dc064a32531" containerName="payload-processing"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587824    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="496d9abc-46c4-417e-8d6a-1dc064a32531" containerName="payload-processing"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587831    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aeff591-42ec-419b-b7cb-bd18d25e0415" containerName="maas-api"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587836    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aeff591-42ec-419b-b7cb-bd18d25e0415" containerName="maas-api"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587844    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ae96c9-a528-499e-98f5-a805cc2ceb6b" containerName="payload-processing"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587849    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ae96c9-a528-499e-98f5-a805cc2ceb6b" containerName="payload-processing"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587856    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413bc16a-520e-4bff-9a33-8a4b06621991" containerName="payload-processing"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587861    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="413bc16a-520e-4bff-9a33-8a4b06621991" containerName="payload-processing"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587873    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fac2a2c-bc1e-4954-bdc7-8c5954143d2a" containerName="maas-api"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587878    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fac2a2c-bc1e-4954-bdc7-8c5954143d2a" containerName="maas-api"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587940    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fac2a2c-bc1e-4954-bdc7-8c5954143d2a" containerName="maas-api"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587949    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="496d9abc-46c4-417e-8d6a-1dc064a32531" containerName="payload-processing"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587954    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aeff591-42ec-419b-b7cb-bd18d25e0415" containerName="maas-api"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587961    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ae96c9-a528-499e-98f5-a805cc2ceb6b" containerName="payload-processing"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587966    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9584210-621b-498f-9467-c6555392a6d4" containerName="main"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587971    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="413bc16a-520e-4bff-9a33-8a4b06621991" containerName="payload-processing"
Sep 06 10:55:11.588011 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.587977    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="faf45f85-c474-404e-ae1e-ad2892a68bbb" containerName="maas-api"
Sep 06 10:55:11.591308 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.591286    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.593761 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.593741    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-605c10c6-kserve-self-signed-certs\""
Sep 06 10:55:11.601661 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.601636    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"]
Sep 06 10:55:11.648555 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.648521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tls-certs\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.648703 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.648559    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tmp-dir\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.648703 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.648587    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-dshm\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.648703 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.648666    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdlg\" (UniqueName: \"kubernetes.io/projected/efb0a7fd-b2b8-42e4-be5b-72c147e44354-kube-api-access-fcdlg\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.648818 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.648718    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-home\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.648818 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.648747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-model-cache\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.749380 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.749345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdlg\" (UniqueName: \"kubernetes.io/projected/efb0a7fd-b2b8-42e4-be5b-72c147e44354-kube-api-access-fcdlg\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.749541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.749391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-home\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.749541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.749437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-model-cache\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.749541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.749495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tls-certs\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.749541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.749522    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tmp-dir\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.749696 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.749552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-dshm\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.749744 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.749725    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-home\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.749865 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.749843    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-model-cache\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.749939 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.749921    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tmp-dir\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.751842 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.751816    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-dshm\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.752082 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.752064    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tls-certs\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.756894 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.756875    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdlg\" (UniqueName: \"kubernetes.io/projected/efb0a7fd-b2b8-42e4-be5b-72c147e44354-kube-api-access-fcdlg\") pod \"e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") " pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:11.904194 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:11.904112    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:12.174340 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:12.174313    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"]
Sep 06 10:55:12.176584 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:55:12.176561    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefb0a7fd_b2b8_42e4_be5b_72c147e44354.slice/crio-30db8693bcab730ea09658850932c19c13edb435d50b511c120478c69f2902c2 WatchSource:0}: Error finding container 30db8693bcab730ea09658850932c19c13edb435d50b511c120478c69f2902c2: Status 404 returned error can't find the container with id 30db8693bcab730ea09658850932c19c13edb435d50b511c120478c69f2902c2
Sep 06 10:55:13.153994 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:13.153958    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96" event={"ID":"efb0a7fd-b2b8-42e4-be5b-72c147e44354","Type":"ContainerStarted","Data":"f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9"}
Sep 06 10:55:13.153994 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:13.154001    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96" event={"ID":"efb0a7fd-b2b8-42e4-be5b-72c147e44354","Type":"ContainerStarted","Data":"30db8693bcab730ea09658850932c19c13edb435d50b511c120478c69f2902c2"}
Sep 06 10:55:13.171422 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:13.171375    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96" podStartSLOduration=2.171358715 podStartE2EDuration="2.171358715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:55:13.170012229 +0000 UTC m=+2001.653512360" watchObservedRunningTime="2026-09-06 10:55:13.171358715 +0000 UTC m=+2001.654858823"
Sep 06 10:55:19.593442 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.593400    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"]
Sep 06 10:55:19.597164 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.597145    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.599590 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.599568    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-605c10c6-kserve-self-signed-certs\""
Sep 06 10:55:19.604733 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.604713    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"]
Sep 06 10:55:19.724412 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.724377    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-home\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.724412 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.724413    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfzz\" (UniqueName: \"kubernetes.io/projected/bd796b29-17fd-473a-bde6-b59e794c5555-kube-api-access-fqfzz\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.724629 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.724454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd796b29-17fd-473a-bde6-b59e794c5555-tls-certs\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.724629 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.724495    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-model-cache\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.724629 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.724567    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-tmp-dir\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.724629 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.724584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-dshm\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.825572 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.825538    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-tmp-dir\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.825572 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.825576    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-dshm\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.825789 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.825600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-home\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.825789 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.825719    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfzz\" (UniqueName: \"kubernetes.io/projected/bd796b29-17fd-473a-bde6-b59e794c5555-kube-api-access-fqfzz\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.825789 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.825770    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd796b29-17fd-473a-bde6-b59e794c5555-tls-certs\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.825902 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.825811    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-model-cache\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.825967 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.825941    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-tmp-dir\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.826027 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.826008    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-home\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.826141 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.826113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-model-cache\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.827881 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.827855    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-dshm\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.828120 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.828103    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd796b29-17fd-473a-bde6-b59e794c5555-tls-certs\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.833003 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.832983    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfzz\" (UniqueName: \"kubernetes.io/projected/bd796b29-17fd-473a-bde6-b59e794c5555-kube-api-access-fqfzz\") pod \"e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") " pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:19.907235 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:19.907125    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:20.181480 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:20.181451    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"]
Sep 06 10:55:20.183618 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:55:20.183591    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd796b29_17fd_473a_bde6_b59e794c5555.slice/crio-98ee2df00d827cd3b26c7b46bf98b8e097bb6284fa8f41bf1183641d53aca766 WatchSource:0}: Error finding container 98ee2df00d827cd3b26c7b46bf98b8e097bb6284fa8f41bf1183641d53aca766: Status 404 returned error can't find the container with id 98ee2df00d827cd3b26c7b46bf98b8e097bb6284fa8f41bf1183641d53aca766
Sep 06 10:55:21.191929 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:21.191896    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph" event={"ID":"bd796b29-17fd-473a-bde6-b59e794c5555","Type":"ContainerStarted","Data":"f80dea981baded715c02dadc7d636f05807847309dd38e89453f4f12453eda73"}
Sep 06 10:55:21.191929 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:21.191942    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph" event={"ID":"bd796b29-17fd-473a-bde6-b59e794c5555","Type":"ContainerStarted","Data":"98ee2df00d827cd3b26c7b46bf98b8e097bb6284fa8f41bf1183641d53aca766"}
Sep 06 10:55:21.209610 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:21.209551    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph" podStartSLOduration=2.209529922 podStartE2EDuration="2.209529922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:55:21.208994522 +0000 UTC m=+2009.692494632" watchObservedRunningTime="2026-09-06 10:55:21.209529922 +0000 UTC m=+2009.693030031"
Sep 06 10:55:21.904277 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:21.904228    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:21.904470 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:21.904296    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:21.917149 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:21.917124    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:22.208168 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:22.208078    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:29.907415 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:29.907375    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:29.907820 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:29.907438    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:29.920134 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:29.920109    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:30.245411 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:30.245329    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:55:44.247768 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:44.247732    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"]
Sep 06 10:55:45.157561 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:45.157525    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"]
Sep 06 10:55:57.269492 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.269461    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"]
Sep 06 10:55:57.272673 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.272657    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.276863 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.276835    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 10:55:57.276863 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.276835    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 10:55:57.277043 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.276900    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-965kn\""
Sep 06 10:55:57.277334 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.277309    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 06 10:55:57.277708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.277686    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 10:55:57.287749 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.287722    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"]
Sep 06 10:55:57.357124 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.357099    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-maas-api-tls\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.357310 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.357128    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pds8r\" (UniqueName: \"kubernetes.io/projected/5866bcd4-4f77-4932-953d-3cff90ad77ed-kube-api-access-pds8r\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.357310 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.357238    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5866bcd4-4f77-4932-953d-3cff90ad77ed-openshift-service-ca\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.357310 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.357280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-metrics-certs\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.458596 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.458571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-maas-api-tls\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.458722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.458604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pds8r\" (UniqueName: \"kubernetes.io/projected/5866bcd4-4f77-4932-953d-3cff90ad77ed-kube-api-access-pds8r\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.458722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.458647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5866bcd4-4f77-4932-953d-3cff90ad77ed-openshift-service-ca\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.458722 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.458675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-metrics-certs\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.459378 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.459359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5866bcd4-4f77-4932-953d-3cff90ad77ed-openshift-service-ca\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.461051 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.461024    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-maas-api-tls\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.461166 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.461149    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-metrics-certs\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.465820 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.465796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pds8r\" (UniqueName: \"kubernetes.io/projected/5866bcd4-4f77-4932-953d-3cff90ad77ed-kube-api-access-pds8r\") pod \"maas-api-679c995ddf-6q9k9\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.587961 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.587930    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:57.859416 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.859269    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"]
Sep 06 10:55:57.861725 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:55:57.861697    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5866bcd4_4f77_4932_953d_3cff90ad77ed.slice/crio-e5c6085c7cccb0721f973cca5a4aa0d1c1d4ed3c83312953f3c758f79d695182 WatchSource:0}: Error finding container e5c6085c7cccb0721f973cca5a4aa0d1c1d4ed3c83312953f3c758f79d695182: Status 404 returned error can't find the container with id e5c6085c7cccb0721f973cca5a4aa0d1c1d4ed3c83312953f3c758f79d695182
Sep 06 10:55:57.863093 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:57.863073    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:55:58.366712 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:58.366673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9" event={"ID":"5866bcd4-4f77-4932-953d-3cff90ad77ed","Type":"ContainerStarted","Data":"901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05"}
Sep 06 10:55:58.367116 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:58.366734    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9" event={"ID":"5866bcd4-4f77-4932-953d-3cff90ad77ed","Type":"ContainerStarted","Data":"e5c6085c7cccb0721f973cca5a4aa0d1c1d4ed3c83312953f3c758f79d695182"}
Sep 06 10:55:58.367116 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:58.366763    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:55:58.383585 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:58.383537    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9" podStartSLOduration=0.968520543 podStartE2EDuration="1.383520043s" totalImagesPullingTime="414.9995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:55:58.38238267 +0000 UTC m=+2046.865882820" watchObservedRunningTime="2026-09-06 10:55:58.383520043 +0000 UTC m=+2046.867020151"
Sep 06 10:55:59.270454 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.270389    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96" podUID="efb0a7fd-b2b8-42e4-be5b-72c147e44354" containerName="main" containerID="cri-o://f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9" gracePeriod=45
Sep 06 10:55:59.523931 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.523871    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:55:59.678550 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.678520    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-model-cache\") pod \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") "
Sep 06 10:55:59.678715 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.678568    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-home\") pod \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") "
Sep 06 10:55:59.678715 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.678612    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-dshm\") pod \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") "
Sep 06 10:55:59.678715 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.678671    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tmp-dir\") pod \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") "
Sep 06 10:55:59.678715 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.678691    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcdlg\" (UniqueName: \"kubernetes.io/projected/efb0a7fd-b2b8-42e4-be5b-72c147e44354-kube-api-access-fcdlg\") pod \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") "
Sep 06 10:55:59.678903 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.678738    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tls-certs\") pod \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\" (UID: \"efb0a7fd-b2b8-42e4-be5b-72c147e44354\") "
Sep 06 10:55:59.678903 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.678779    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-model-cache" (OuterVolumeSpecName: "model-cache") pod "efb0a7fd-b2b8-42e4-be5b-72c147e44354" (UID: "efb0a7fd-b2b8-42e4-be5b-72c147e44354"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:55:59.678903 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.678814    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-home" (OuterVolumeSpecName: "home") pod "efb0a7fd-b2b8-42e4-be5b-72c147e44354" (UID: "efb0a7fd-b2b8-42e4-be5b-72c147e44354"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:55:59.679072 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.678947    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efb0a7fd-b2b8-42e4-be5b-72c147e44354" (UID: "efb0a7fd-b2b8-42e4-be5b-72c147e44354"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:55:59.679072 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.679006    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:55:59.679072 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.679025    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:55:59.679072 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.679037    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:55:59.680853 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.680824    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-dshm" (OuterVolumeSpecName: "dshm") pod "efb0a7fd-b2b8-42e4-be5b-72c147e44354" (UID: "efb0a7fd-b2b8-42e4-be5b-72c147e44354"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:55:59.680983 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.680906    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efb0a7fd-b2b8-42e4-be5b-72c147e44354" (UID: "efb0a7fd-b2b8-42e4-be5b-72c147e44354"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:55:59.680983 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.680906    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb0a7fd-b2b8-42e4-be5b-72c147e44354-kube-api-access-fcdlg" (OuterVolumeSpecName: "kube-api-access-fcdlg") pod "efb0a7fd-b2b8-42e4-be5b-72c147e44354" (UID: "efb0a7fd-b2b8-42e4-be5b-72c147e44354"). InnerVolumeSpecName "kube-api-access-fcdlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:55:59.779628 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.779553    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb0a7fd-b2b8-42e4-be5b-72c147e44354-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:55:59.779628 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.779578    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb0a7fd-b2b8-42e4-be5b-72c147e44354-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:55:59.779628 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:55:59.779588    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcdlg\" (UniqueName: \"kubernetes.io/projected/efb0a7fd-b2b8-42e4-be5b-72c147e44354-kube-api-access-fcdlg\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:00.177237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.177184    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph" podUID="bd796b29-17fd-473a-bde6-b59e794c5555" containerName="main" containerID="cri-o://f80dea981baded715c02dadc7d636f05807847309dd38e89453f4f12453eda73" gracePeriod=45
Sep 06 10:56:00.235416 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.235384    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph" podUID="bd796b29-17fd-473a-bde6-b59e794c5555" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 06 10:56:00.378113 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.378079    2564 generic.go:356] "Generic (PLEG): container finished" podID="bd796b29-17fd-473a-bde6-b59e794c5555" containerID="f80dea981baded715c02dadc7d636f05807847309dd38e89453f4f12453eda73" exitCode=0
Sep 06 10:56:00.378286 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.378152    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph" event={"ID":"bd796b29-17fd-473a-bde6-b59e794c5555","Type":"ContainerDied","Data":"f80dea981baded715c02dadc7d636f05807847309dd38e89453f4f12453eda73"}
Sep 06 10:56:00.379509 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.379485    2564 generic.go:356] "Generic (PLEG): container finished" podID="efb0a7fd-b2b8-42e4-be5b-72c147e44354" containerID="f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9" exitCode=0
Sep 06 10:56:00.379631 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.379518    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96" event={"ID":"efb0a7fd-b2b8-42e4-be5b-72c147e44354","Type":"ContainerDied","Data":"f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9"}
Sep 06 10:56:00.379631 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.379538    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96" event={"ID":"efb0a7fd-b2b8-42e4-be5b-72c147e44354","Type":"ContainerDied","Data":"30db8693bcab730ea09658850932c19c13edb435d50b511c120478c69f2902c2"}
Sep 06 10:56:00.379631 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.379540    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"
Sep 06 10:56:00.379631 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.379558    2564 scope.go:117] "RemoveContainer" containerID="f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9"
Sep 06 10:56:00.388083 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.388064    2564 scope.go:117] "RemoveContainer" containerID="f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9"
Sep 06 10:56:00.388363 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:56:00.388340    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9\": container with ID starting with f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9 not found: ID does not exist" containerID="f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9"
Sep 06 10:56:00.388485 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.388367    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9"} err="failed to get container status \"f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9\": rpc error: code = NotFound desc = could not find container \"f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9\": container with ID starting with f501891385aad12bfcd79e372aa000e783134a501344e5a22ebf7e8c962cc3e9 not found: ID does not exist"
Sep 06 10:56:00.394745 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.394723    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"]
Sep 06 10:56:00.398738 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.398719    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-605c10c6-kserve-6b44bb5c59-fcq96"]
Sep 06 10:56:00.407599 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.407583    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:56:00.485654 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.485624    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd796b29-17fd-473a-bde6-b59e794c5555-tls-certs\") pod \"bd796b29-17fd-473a-bde6-b59e794c5555\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") "
Sep 06 10:56:00.485801 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.485664    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-tmp-dir\") pod \"bd796b29-17fd-473a-bde6-b59e794c5555\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") "
Sep 06 10:56:00.485801 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.485698    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqfzz\" (UniqueName: \"kubernetes.io/projected/bd796b29-17fd-473a-bde6-b59e794c5555-kube-api-access-fqfzz\") pod \"bd796b29-17fd-473a-bde6-b59e794c5555\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") "
Sep 06 10:56:00.485801 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.485762    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-home\") pod \"bd796b29-17fd-473a-bde6-b59e794c5555\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") "
Sep 06 10:56:00.485801 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.485784    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-dshm\") pod \"bd796b29-17fd-473a-bde6-b59e794c5555\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") "
Sep 06 10:56:00.486035 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.485847    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-model-cache\") pod \"bd796b29-17fd-473a-bde6-b59e794c5555\" (UID: \"bd796b29-17fd-473a-bde6-b59e794c5555\") "
Sep 06 10:56:00.486035 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.485886    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd796b29-17fd-473a-bde6-b59e794c5555" (UID: "bd796b29-17fd-473a-bde6-b59e794c5555"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:56:00.486125 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.486022    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-home" (OuterVolumeSpecName: "home") pod "bd796b29-17fd-473a-bde6-b59e794c5555" (UID: "bd796b29-17fd-473a-bde6-b59e794c5555"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:56:00.486125 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.486077    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd796b29-17fd-473a-bde6-b59e794c5555" (UID: "bd796b29-17fd-473a-bde6-b59e794c5555"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:56:00.486204 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.486186    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:00.486261 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.486203    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:00.486261 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.486234    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:00.487895 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.487875    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd796b29-17fd-473a-bde6-b59e794c5555-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd796b29-17fd-473a-bde6-b59e794c5555" (UID: "bd796b29-17fd-473a-bde6-b59e794c5555"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:56:00.487960 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.487896    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-dshm" (OuterVolumeSpecName: "dshm") pod "bd796b29-17fd-473a-bde6-b59e794c5555" (UID: "bd796b29-17fd-473a-bde6-b59e794c5555"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:56:00.487960 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.487909    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd796b29-17fd-473a-bde6-b59e794c5555-kube-api-access-fqfzz" (OuterVolumeSpecName: "kube-api-access-fqfzz") pod "bd796b29-17fd-473a-bde6-b59e794c5555" (UID: "bd796b29-17fd-473a-bde6-b59e794c5555"). InnerVolumeSpecName "kube-api-access-fqfzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:56:00.587270 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.587184    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd796b29-17fd-473a-bde6-b59e794c5555-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:00.587270 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.587237    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfzz\" (UniqueName: \"kubernetes.io/projected/bd796b29-17fd-473a-bde6-b59e794c5555-kube-api-access-fqfzz\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:00.587270 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:00.587247    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd796b29-17fd-473a-bde6-b59e794c5555-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:01.392552 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:01.392513    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph" event={"ID":"bd796b29-17fd-473a-bde6-b59e794c5555","Type":"ContainerDied","Data":"98ee2df00d827cd3b26c7b46bf98b8e097bb6284fa8f41bf1183641d53aca766"}
Sep 06 10:56:01.392730 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:01.392568    2564 scope.go:117] "RemoveContainer" containerID="f80dea981baded715c02dadc7d636f05807847309dd38e89453f4f12453eda73"
Sep 06 10:56:01.392730 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:01.392535    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"
Sep 06 10:56:01.413823 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:01.413798    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"]
Sep 06 10:56:01.417568 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:01.417546    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-605c10c6-kserve-5bf95f847-9rzph"]
Sep 06 10:56:02.030055 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:02.030023    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd796b29-17fd-473a-bde6-b59e794c5555" path="/var/lib/kubelet/pods/bd796b29-17fd-473a-bde6-b59e794c5555/volumes"
Sep 06 10:56:02.030454 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:02.030408    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb0a7fd-b2b8-42e4-be5b-72c147e44354" path="/var/lib/kubelet/pods/efb0a7fd-b2b8-42e4-be5b-72c147e44354/volumes"
Sep 06 10:56:04.377103 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:04.377075    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:56:04.943099 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:04.943062    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"]
Sep 06 10:56:04.943523 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:04.943504    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd796b29-17fd-473a-bde6-b59e794c5555" containerName="main"
Sep 06 10:56:04.943523 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:04.943525    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd796b29-17fd-473a-bde6-b59e794c5555" containerName="main"
Sep 06 10:56:04.943689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:04.943554    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb0a7fd-b2b8-42e4-be5b-72c147e44354" containerName="main"
Sep 06 10:56:04.943689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:04.943562    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb0a7fd-b2b8-42e4-be5b-72c147e44354" containerName="main"
Sep 06 10:56:04.943689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:04.943655    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd796b29-17fd-473a-bde6-b59e794c5555" containerName="main"
Sep 06 10:56:04.943689 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:04.943669    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb0a7fd-b2b8-42e4-be5b-72c147e44354" containerName="main"
Sep 06 10:56:04.948010 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:04.947987    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:04.957239 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:04.957202    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"]
Sep 06 10:56:05.027776 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.027748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a694d4-3b44-402e-818c-c2598baa5779-metrics-certs\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.027776 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.027779    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgg49\" (UniqueName: \"kubernetes.io/projected/04a694d4-3b44-402e-818c-c2598baa5779-kube-api-access-cgg49\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.027976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.027804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a694d4-3b44-402e-818c-c2598baa5779-openshift-service-ca\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.027976 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.027876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a694d4-3b44-402e-818c-c2598baa5779-maas-api-tls\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.128768 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.128737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a694d4-3b44-402e-818c-c2598baa5779-maas-api-tls\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.128924 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.128828    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a694d4-3b44-402e-818c-c2598baa5779-metrics-certs\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.128924 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.128854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgg49\" (UniqueName: \"kubernetes.io/projected/04a694d4-3b44-402e-818c-c2598baa5779-kube-api-access-cgg49\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.128924 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.128887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a694d4-3b44-402e-818c-c2598baa5779-openshift-service-ca\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.129578 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.129556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a694d4-3b44-402e-818c-c2598baa5779-openshift-service-ca\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.131229 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.131195    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a694d4-3b44-402e-818c-c2598baa5779-maas-api-tls\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.131428 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.131408    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a694d4-3b44-402e-818c-c2598baa5779-metrics-certs\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.136977 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.136951    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgg49\" (UniqueName: \"kubernetes.io/projected/04a694d4-3b44-402e-818c-c2598baa5779-kube-api-access-cgg49\") pod \"maas-api-58fd75565c-qf89p\" (UID: \"04a694d4-3b44-402e-818c-c2598baa5779\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.259692 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.259624    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:05.534819 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:05.534793    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"]
Sep 06 10:56:05.536743 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:56:05.536715    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a694d4_3b44_402e_818c_c2598baa5779.slice/crio-187e02729d5630041f1a992692c2e28d7c8ce980fe236363ddf05dfceb6f9f4a WatchSource:0}: Error finding container 187e02729d5630041f1a992692c2e28d7c8ce980fe236363ddf05dfceb6f9f4a: Status 404 returned error can't find the container with id 187e02729d5630041f1a992692c2e28d7c8ce980fe236363ddf05dfceb6f9f4a
Sep 06 10:56:06.420363 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:06.420330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p" event={"ID":"04a694d4-3b44-402e-818c-c2598baa5779","Type":"ContainerStarted","Data":"e8cfb629a7cd9650100f89907e60c899f3b013e48bab80b9c1b0e1a2f6a64bbb"}
Sep 06 10:56:06.420543 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:06.420381    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p" event={"ID":"04a694d4-3b44-402e-818c-c2598baa5779","Type":"ContainerStarted","Data":"187e02729d5630041f1a992692c2e28d7c8ce980fe236363ddf05dfceb6f9f4a"}
Sep 06 10:56:06.420543 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:06.420432    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:06.438604 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:06.438562    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p" podStartSLOduration=2.111030025 podStartE2EDuration="2.438545886s" totalImagesPullingTime="327.515861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:56:06.436841784 +0000 UTC m=+2054.920341893" watchObservedRunningTime="2026-09-06 10:56:06.438545886 +0000 UTC m=+2054.922045996"
Sep 06 10:56:12.430964 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:12.430934    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58fd75565c-qf89p"
Sep 06 10:56:12.478525 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:12.478495    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"]
Sep 06 10:56:12.478746 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:12.478722    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9" podUID="5866bcd4-4f77-4932-953d-3cff90ad77ed" containerName="maas-api" containerID="cri-o://901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05" gracePeriod=30
Sep 06 10:56:13.726504 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:13.726482    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:56:13.908488 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:13.908385    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-maas-api-tls\") pod \"5866bcd4-4f77-4932-953d-3cff90ad77ed\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") "
Sep 06 10:56:13.908488 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:13.908448    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5866bcd4-4f77-4932-953d-3cff90ad77ed-openshift-service-ca\") pod \"5866bcd4-4f77-4932-953d-3cff90ad77ed\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") "
Sep 06 10:56:13.908488 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:13.908489    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-metrics-certs\") pod \"5866bcd4-4f77-4932-953d-3cff90ad77ed\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") "
Sep 06 10:56:13.908830 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:13.908647    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pds8r\" (UniqueName: \"kubernetes.io/projected/5866bcd4-4f77-4932-953d-3cff90ad77ed-kube-api-access-pds8r\") pod \"5866bcd4-4f77-4932-953d-3cff90ad77ed\" (UID: \"5866bcd4-4f77-4932-953d-3cff90ad77ed\") "
Sep 06 10:56:13.908881 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:13.908820    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5866bcd4-4f77-4932-953d-3cff90ad77ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5866bcd4-4f77-4932-953d-3cff90ad77ed" (UID: "5866bcd4-4f77-4932-953d-3cff90ad77ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:56:13.909042 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:13.909021    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5866bcd4-4f77-4932-953d-3cff90ad77ed-openshift-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:13.910682 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:13.910657    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5866bcd4-4f77-4932-953d-3cff90ad77ed-kube-api-access-pds8r" (OuterVolumeSpecName: "kube-api-access-pds8r") pod "5866bcd4-4f77-4932-953d-3cff90ad77ed" (UID: "5866bcd4-4f77-4932-953d-3cff90ad77ed"). InnerVolumeSpecName "kube-api-access-pds8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:56:13.910682 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:13.910673    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5866bcd4-4f77-4932-953d-3cff90ad77ed" (UID: "5866bcd4-4f77-4932-953d-3cff90ad77ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:56:13.910806 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:13.910719    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5866bcd4-4f77-4932-953d-3cff90ad77ed" (UID: "5866bcd4-4f77-4932-953d-3cff90ad77ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:56:14.010542 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.010490    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pds8r\" (UniqueName: \"kubernetes.io/projected/5866bcd4-4f77-4932-953d-3cff90ad77ed-kube-api-access-pds8r\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:14.010542 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.010536    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-maas-api-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:14.010542 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.010545    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5866bcd4-4f77-4932-953d-3cff90ad77ed-metrics-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\""
Sep 06 10:56:14.466994 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.466958    2564 generic.go:356] "Generic (PLEG): container finished" podID="5866bcd4-4f77-4932-953d-3cff90ad77ed" containerID="901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05" exitCode=0
Sep 06 10:56:14.467227 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.467006    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"
Sep 06 10:56:14.467227 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.467038    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9" event={"ID":"5866bcd4-4f77-4932-953d-3cff90ad77ed","Type":"ContainerDied","Data":"901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05"}
Sep 06 10:56:14.467227 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.467104    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9" event={"ID":"5866bcd4-4f77-4932-953d-3cff90ad77ed","Type":"ContainerDied","Data":"e5c6085c7cccb0721f973cca5a4aa0d1c1d4ed3c83312953f3c758f79d695182"}
Sep 06 10:56:14.467227 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.467129    2564 scope.go:117] "RemoveContainer" containerID="901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05"
Sep 06 10:56:14.476075 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.476051    2564 scope.go:117] "RemoveContainer" containerID="901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05"
Sep 06 10:56:14.476390 ip-10-0-130-112 kubenswrapper[2564]: E0906 10:56:14.476368    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05\": container with ID starting with 901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05 not found: ID does not exist" containerID="901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05"
Sep 06 10:56:14.476449 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.476402    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05"} err="failed to get container status \"901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05\": rpc error: code = NotFound desc = could not find container \"901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05\": container with ID starting with 901baec5953bf32ca2d7d23a818ec732b893f6e19a62b9689c3a8bcdc4a36c05 not found: ID does not exist"
Sep 06 10:56:14.483122 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.483101    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"]
Sep 06 10:56:14.486451 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:14.486431    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-6q9k9"]
Sep 06 10:56:16.028371 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:16.028330    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5866bcd4-4f77-4932-953d-3cff90ad77ed" path="/var/lib/kubelet/pods/5866bcd4-4f77-4932-953d-3cff90ad77ed/volumes"
Sep 06 10:56:30.379984 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:30.379951    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79b78b5455-djpqp_a9691cd0-c563-4bd8-8ecc-82bf98234a9d/authorino/0.log"
Sep 06 10:56:36.513201 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:36.513169    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c688cbfb6-g7qgl_def6d513-734b-4979-b74e-ab5dbda86afe/copy-manifests/0.log"
Sep 06 10:56:36.518516 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:36.518493    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c688cbfb6-g7qgl_def6d513-734b-4979-b74e-ab5dbda86afe/manager/0.log"
Sep 06 10:56:37.118298 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:37.118270    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58fd75565c-qf89p_04a694d4-3b44-402e-818c-c2598baa5779/maas-api/0.log"
Sep 06 10:56:38.131394 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:38.127832    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79b78b5455-djpqp_a9691cd0-c563-4bd8-8ecc-82bf98234a9d/authorino/0.log"
Sep 06 10:56:38.464265 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:38.464172    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-rpk2v_0b321129-77f2-4396-9c55-22234b9f1847/kuadrant-console-plugin/0.log"
Sep 06 10:56:38.839376 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:38.839345    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-7xwwq_edf3481b-d9d7-48f8-8c51-3d6b4140abf3/manager/0.log"
Sep 06 10:56:39.533406 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:39.533378    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f5945d9f-m9pss_ba008a94-cc59-4b58-b4f8-7833f5cefc42/kube-auth-proxy/0.log"
Sep 06 10:56:40.737205 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:40.737172    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4bbsn_e9f5c2e8-d415-438b-bc6f-d7b7b506a3e3/main/0.log"
Sep 06 10:56:41.074092 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:41.074065    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tgvc6_a72503ff-78e2-4ef8-8cef-e88933f62c94/main/0.log"
Sep 06 10:56:49.336071 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:49.336046    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hpfqd_6a17bb93-98a4-4d0b-8f03-2c86b1f09b0d/global-pull-secret-syncer/0.log"
Sep 06 10:56:49.483602 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:49.483571    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qk4m4_b01a3bd2-a82b-43e9-81b7-f94eeede7fd5/konnectivity-agent/0.log"
Sep 06 10:56:49.526581 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:49.526556    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-112.ec2.internal_02cbb65cfbfc76887ba93e94ab94587f/haproxy/0.log"
Sep 06 10:56:52.112426 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:52.112399    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:56:52.123622 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:52.123601    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bsdqh_c7e1f1e1-44aa-4007-a88d-58ad1c176394/ovn-acl-logging/0.log"
Sep 06 10:56:54.602178 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:54.602148    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79b78b5455-djpqp_a9691cd0-c563-4bd8-8ecc-82bf98234a9d/authorino/0.log"
Sep 06 10:56:54.689988 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:54.689964    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-rpk2v_0b321129-77f2-4396-9c55-22234b9f1847/kuadrant-console-plugin/0.log"
Sep 06 10:56:54.813763 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:54.813737    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-7xwwq_edf3481b-d9d7-48f8-8c51-3d6b4140abf3/manager/0.log"
Sep 06 10:56:56.403016 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.402986    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-brshs_48d31127-0fb1-4758-91b6-26f51b39b5c8/kube-state-metrics/0.log"
Sep 06 10:56:56.420028 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.420001    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-brshs_48d31127-0fb1-4758-91b6-26f51b39b5c8/kube-rbac-proxy-main/0.log"
Sep 06 10:56:56.441652 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.441633    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-brshs_48d31127-0fb1-4758-91b6-26f51b39b5c8/kube-rbac-proxy-self/0.log"
Sep 06 10:56:56.481929 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.481911    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6c5874989c-ddjz6_09208b3f-f2a1-450c-a747-5316c8296be3/metrics-server/0.log"
Sep 06 10:56:56.523455 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.523438    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-rcdvj_9a981f69-c807-4a4e-85a6-0f8deb6f5e31/monitoring-plugin/0.log"
Sep 06 10:56:56.666382 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.666314    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n72xw_a91dff9a-5731-4beb-93a8-a0f6efb80dad/node-exporter/0.log"
Sep 06 10:56:56.694745 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.694717    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n72xw_a91dff9a-5731-4beb-93a8-a0f6efb80dad/kube-rbac-proxy/0.log"
Sep 06 10:56:56.716409 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.716388    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n72xw_a91dff9a-5731-4beb-93a8-a0f6efb80dad/init-textfile/0.log"
Sep 06 10:56:56.820285 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.820263    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pwwkl_250c73c6-2b37-4b8c-90a7-5f8ae4c23c93/kube-rbac-proxy-main/0.log"
Sep 06 10:56:56.838660 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.838638    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pwwkl_250c73c6-2b37-4b8c-90a7-5f8ae4c23c93/kube-rbac-proxy-self/0.log"
Sep 06 10:56:56.858220 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.858195    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pwwkl_250c73c6-2b37-4b8c-90a7-5f8ae4c23c93/openshift-state-metrics/0.log"
Sep 06 10:56:56.899738 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.899716    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ab6bb49a-fd68-4a39-8c03-83646f02a6f1/prometheus/0.log"
Sep 06 10:56:56.922061 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.922009    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ab6bb49a-fd68-4a39-8c03-83646f02a6f1/config-reloader/0.log"
Sep 06 10:56:56.942735 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.942717    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ab6bb49a-fd68-4a39-8c03-83646f02a6f1/thanos-sidecar/0.log"
Sep 06 10:56:56.964483 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:56.964461    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ab6bb49a-fd68-4a39-8c03-83646f02a6f1/kube-rbac-proxy-web/0.log"
Sep 06 10:56:57.001243 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:57.001207    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ab6bb49a-fd68-4a39-8c03-83646f02a6f1/kube-rbac-proxy/0.log"
Sep 06 10:56:57.023235 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:57.023205    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ab6bb49a-fd68-4a39-8c03-83646f02a6f1/kube-rbac-proxy-thanos/0.log"
Sep 06 10:56:57.055388 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:57.055371    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ab6bb49a-fd68-4a39-8c03-83646f02a6f1/init-config-reloader/0.log"
Sep 06 10:56:57.211836 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:57.211774    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-bjmmw_0ed020c3-6474-434e-b11d-c0ecb7f68e83/prometheus-operator-admission-webhook/0.log"
Sep 06 10:56:57.330448 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:57.330423    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d9844b69-9w62x_d6c3cfad-e5b4-4380-9c1e-286cc8dd1373/thanos-query/0.log"
Sep 06 10:56:57.349481 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:57.349462    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d9844b69-9w62x_d6c3cfad-e5b4-4380-9c1e-286cc8dd1373/kube-rbac-proxy-web/0.log"
Sep 06 10:56:57.368951 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:57.368931    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d9844b69-9w62x_d6c3cfad-e5b4-4380-9c1e-286cc8dd1373/kube-rbac-proxy/0.log"
Sep 06 10:56:57.386191 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:57.386169    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d9844b69-9w62x_d6c3cfad-e5b4-4380-9c1e-286cc8dd1373/prom-label-proxy/0.log"
Sep 06 10:56:57.404681 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:57.404657    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d9844b69-9w62x_d6c3cfad-e5b4-4380-9c1e-286cc8dd1373/kube-rbac-proxy-rules/0.log"
Sep 06 10:56:57.426018 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:57.425996    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56d9844b69-9w62x_d6c3cfad-e5b4-4380-9c1e-286cc8dd1373/kube-rbac-proxy-metrics/0.log"
Sep 06 10:56:59.350773 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.350741    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"]
Sep 06 10:56:59.351141 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.351108    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5866bcd4-4f77-4932-953d-3cff90ad77ed" containerName="maas-api"
Sep 06 10:56:59.351141 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.351122    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5866bcd4-4f77-4932-953d-3cff90ad77ed" containerName="maas-api"
Sep 06 10:56:59.351237 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.351199    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5866bcd4-4f77-4932-953d-3cff90ad77ed" containerName="maas-api"
Sep 06 10:56:59.354291 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.354273    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.356803 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.356775    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qdn88\"/\"openshift-service-ca.crt\""
Sep 06 10:56:59.356902 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.356809    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qdn88\"/\"kube-root-ca.crt\""
Sep 06 10:56:59.357701 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.357683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qdn88\"/\"default-dockercfg-lbxvv\""
Sep 06 10:56:59.361964 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.361923    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"]
Sep 06 10:56:59.424265 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.424241    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-sys\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.424368 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.424322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-lib-modules\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.424368 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.424351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4wq\" (UniqueName: \"kubernetes.io/projected/9490fcdc-45af-4f81-aae1-2bb5b6a733be-kube-api-access-7z4wq\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.424444 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.424398    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-proc\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.424444 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.424433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-podres\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.525492 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.525462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-lib-modules\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.525622 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.525497    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4wq\" (UniqueName: \"kubernetes.io/projected/9490fcdc-45af-4f81-aae1-2bb5b6a733be-kube-api-access-7z4wq\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.525622 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.525524    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-proc\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.525622 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.525542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-podres\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.525622 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.525587    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-sys\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.525786 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.525624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-proc\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.525786 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.525643    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-lib-modules\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.525786 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.525663    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-sys\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.525786 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.525730    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9490fcdc-45af-4f81-aae1-2bb5b6a733be-podres\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.533802 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.533772    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4wq\" (UniqueName: \"kubernetes.io/projected/9490fcdc-45af-4f81-aae1-2bb5b6a733be-kube-api-access-7z4wq\") pod \"perf-node-gather-daemonset-kdr2c\" (UID: \"9490fcdc-45af-4f81-aae1-2bb5b6a733be\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.665724 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.665659    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:56:59.936041 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.936004    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"]
Sep 06 10:56:59.938646 ip-10-0-130-112 kubenswrapper[2564]: W0906 10:56:59.938621    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9490fcdc_45af_4f81_aae1_2bb5b6a733be.slice/crio-d779c2cc2e97bc3427fc9b02ae801448103c6cdd8d3bafedcc8285a9674b6cce WatchSource:0}: Error finding container d779c2cc2e97bc3427fc9b02ae801448103c6cdd8d3bafedcc8285a9674b6cce: Status 404 returned error can't find the container with id d779c2cc2e97bc3427fc9b02ae801448103c6cdd8d3bafedcc8285a9674b6cce
Sep 06 10:56:59.966354 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:56:59.966320    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-h8bhl_2549c291-d66f-4887-93ce-cc0ad7d761f7/download-server/0.log"
Sep 06 10:57:00.685403 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:00.685372    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c" event={"ID":"9490fcdc-45af-4f81-aae1-2bb5b6a733be","Type":"ContainerStarted","Data":"352927a50d56480b9712bb525c86b9e0db0ed82881652e22b87ec6589fd97bef"}
Sep 06 10:57:00.685811 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:00.685422    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c" event={"ID":"9490fcdc-45af-4f81-aae1-2bb5b6a733be","Type":"ContainerStarted","Data":"d779c2cc2e97bc3427fc9b02ae801448103c6cdd8d3bafedcc8285a9674b6cce"}
Sep 06 10:57:00.685811 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:00.685479    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:57:00.706508 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:00.700677    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c" podStartSLOduration=1.700659553 podStartE2EDuration="1.700659553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:57:00.699499379 +0000 UTC m=+2109.182999520" watchObservedRunningTime="2026-09-06 10:57:00.700659553 +0000 UTC m=+2109.184159663"
Sep 06 10:57:01.719794 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:01.719769    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cbr79_3e262594-dced-40d6-b678-a0d04be6252b/dns/0.log"
Sep 06 10:57:01.739794 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:01.739766    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cbr79_3e262594-dced-40d6-b678-a0d04be6252b/kube-rbac-proxy/0.log"
Sep 06 10:57:01.843098 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:01.843068    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mjk44_7e2d842b-52a5-4065-a84b-1e8b047e2ba7/dns-node-resolver/0.log"
Sep 06 10:57:02.396419 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:02.396392    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f64dddb67-t4fz6_71bd4b2d-5317-4a86-9bbf-9995f3b667f2/registry/0.log"
Sep 06 10:57:02.416590 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:02.416564    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5wxtv_7ad43d1d-8428-4eae-a728-b9052f618103/node-ca/0.log"
Sep 06 10:57:03.755424 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:03.755397    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f5945d9f-m9pss_ba008a94-cc59-4b58-b4f8-7833f5cefc42/kube-auth-proxy/0.log"
Sep 06 10:57:04.752053 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:04.752027    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7fpmw_7d63a459-955c-424e-ad42-e5d112a2e6d9/serve-healthcheck-canary/0.log"
Sep 06 10:57:05.292667 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:05.292640    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-25jrg_4c23f4e2-c759-4b6a-af25-62e437473b3a/kube-rbac-proxy/0.log"
Sep 06 10:57:05.310588 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:05.310564    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-25jrg_4c23f4e2-c759-4b6a-af25-62e437473b3a/exporter/0.log"
Sep 06 10:57:05.329822 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:05.329800    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-25jrg_4c23f4e2-c759-4b6a-af25-62e437473b3a/extractor/0.log"
Sep 06 10:57:06.700593 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:06.700568    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-kdr2c"
Sep 06 10:57:08.123770 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:08.123743    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c688cbfb6-g7qgl_def6d513-734b-4979-b74e-ab5dbda86afe/manager/0.log"
Sep 06 10:57:08.142576 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:08.142553    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c688cbfb6-g7qgl_def6d513-734b-4979-b74e-ab5dbda86afe/copy-manifests/0.log"
Sep 06 10:57:09.681501 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:09.681472    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5dc6f584c9-hz2fs_52be845f-2e4a-489a-a926-82f33592a6ca/manager/0.log"
Sep 06 10:57:10.567803 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:10.567771    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f5945d9f-m9pss_ba008a94-cc59-4b58-b4f8-7833f5cefc42/kube-auth-proxy/0.log"
Sep 06 10:57:12.440994 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:12.440956    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f5945d9f-m9pss_ba008a94-cc59-4b58-b4f8-7833f5cefc42/kube-auth-proxy/0.log"
Sep 06 10:57:14.670476 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:14.670446    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-86x2h_31d7c255-86b6-45b2-a235-577e6b069272/migrator/0.log"
Sep 06 10:57:14.687708 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:14.687685    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-86x2h_31d7c255-86b6-45b2-a235-577e6b069272/graceful-termination/0.log"
Sep 06 10:57:16.279347 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:16.279319    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-668b9_7fa465d8-bbd3-4102-ae4f-7bba5685b27d/kube-multus-additional-cni-plugins/0.log"
Sep 06 10:57:16.298314 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:16.298289    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-668b9_7fa465d8-bbd3-4102-ae4f-7bba5685b27d/egress-router-binary-copy/0.log"
Sep 06 10:57:16.317719 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:16.317693    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-668b9_7fa465d8-bbd3-4102-ae4f-7bba5685b27d/cni-plugins/0.log"
Sep 06 10:57:16.356542 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:16.356511    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-668b9_7fa465d8-bbd3-4102-ae4f-7bba5685b27d/bond-cni-plugin/0.log"
Sep 06 10:57:16.419541 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:16.419513    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-668b9_7fa465d8-bbd3-4102-ae4f-7bba5685b27d/routeoverride-cni/0.log"
Sep 06 10:57:16.474432 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:16.474404    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-668b9_7fa465d8-bbd3-4102-ae4f-7bba5685b27d/whereabouts-cni-bincopy/0.log"
Sep 06 10:57:16.517358 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:16.517332    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-668b9_7fa465d8-bbd3-4102-ae4f-7bba5685b27d/whereabouts-cni/0.log"
Sep 06 10:57:16.715991 ip-10-0-130-112 kubenswrapper[2564]: I0906 10:57:16.715962    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hj4t9_198a5f6a-7c03-42ab-9a85-1efd3f6e958a/kube-multus/0.log"