Sep 17 14:05:02.289740 ip-10-0-131-146 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 14:05:02.730982 ip-10-0-131-146 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 14:05:02.730982 ip-10-0-131-146 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 14:05:02.730982 ip-10-0-131-146 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 14:05:02.730982 ip-10-0-131-146 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 14:05:02.730982 ip-10-0-131-146 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 14:05:02.732707 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.732603    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 14:05:02.741227 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741194    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:05:02.741227 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741222    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:05:02.741227 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741226    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:05:02.741227 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741230    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:05:02.741227 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741233    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:05:02.741227 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741236    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:05:02.741227 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741239    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741243    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741246    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741249    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741253    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741255    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741258    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741261    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741263    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741267    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741269    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741272    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741274    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741277    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741280    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741283    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741286    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741288    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741291    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741294    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:05:02.741537 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741297    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741299    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741302    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741305    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741307    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741310    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741313    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741316    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741319    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741321    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741324    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741327    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741330    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741332    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741336    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741339    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741342    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741344    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741347    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741349    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:05:02.742053 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741352    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741375    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741378    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741380    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741390    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741393    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741396    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741398    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741401    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741404    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741406    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741409    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741412    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741414    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741417    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741420    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741422    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741425    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741428    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741431    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:05:02.742579 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741433    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741435    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741438    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741441    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741443    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741446    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741449    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741452    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741455    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741457    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741460    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741463    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741465    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741468    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741471    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741474    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741476    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741480    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741482    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:05:02.743056 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741485    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741488    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741492    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741495    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741498    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741501    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741503    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741506    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741509    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741512    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741515    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741519    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741523    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.741526    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742269    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742277    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742281    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742284    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742287    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:05:02.743529 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742291    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742294    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742297    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742300    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742303    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742306    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742309    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742314    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742327    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742331    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742334    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742337    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742340    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742343    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742346    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742349    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742351    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742367    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742370    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:05:02.743994 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742375    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742379    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742382    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742385    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742388    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742390    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742393    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742396    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742398    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742401    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742404    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742406    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742410    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742414    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742417    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742419    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742422    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742425    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742427    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742430    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:05:02.744521 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742432    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742435    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742438    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742440    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742443    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742446    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742449    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742452    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742454    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742457    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742460    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742462    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742465    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742467    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742470    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742472    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742475    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742478    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742480    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742483    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742486    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:05:02.745012 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742488    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742493    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742497    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742499    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742502    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742505    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742508    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742510    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742513    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742516    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742519    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742521    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742524    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742526    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742529    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742532    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742534    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742537    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742540    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:05:02.745546 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742543    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742546    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742548    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742551    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742553    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742556    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742558    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742561    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742564    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742567    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742570    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742572    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742575    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742578    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.742581    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743021    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743027    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743030    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743033    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:05:02.746017 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743036    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743039    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743041    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743044    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743047    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743050    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743052    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743055    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743058    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743061    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743063    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743066    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743069    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743072    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743075    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743078    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743080    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743083    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743086    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:05:02.746502 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743088    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743093    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743097    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743101    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743104    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743107    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743109    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743112    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743115    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743117    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743120    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743122    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743125    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743127    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743130    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743132    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743135    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743137    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743140    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743142    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:05:02.746965 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743145    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743148    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743150    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743153    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743155    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743158    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743161    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743164    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743167    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743170    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743172    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743175    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743177    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743183    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743186    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743188    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743191    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743194    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743197    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:05:02.747480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743199    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743202    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743204    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743207    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743210    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743212    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743215    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743218    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743220    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743223    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743227    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743230    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743232    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743237    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743240    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743242    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743245    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743247    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743250    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743253    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:05:02.747945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743261    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743264    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743266    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743269    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743272    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743274    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743279    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743282    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743285    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743289    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743292    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743295    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743298    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743300    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743303    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743305    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.743308    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743409    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743455    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743468    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 14:05:02.748501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743472    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743477    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743481    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743485    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743490    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743493    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743496    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743500    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743503    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743506    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743509    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743512    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743515    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743520    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743523    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743529    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743532    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743538    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743542    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743546    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743551    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743554    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743557    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 14:05:02.748984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743561    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743564    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743567    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743570    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743573    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743576    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743580    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743584    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743587    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743590    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743593    2560 flags.go:64] FLAG: --enable-server="true"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743596    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743601    2560 flags.go:64] FLAG: --event-burst="100"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743605    2560 flags.go:64] FLAG: --event-qps="50"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743607    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743611    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743614    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743618    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743621    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743624    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743627    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743630    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743633    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743637    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743640    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 14:05:02.749556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743643    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743648    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743652    2560 flags.go:64] FLAG: --feature-gates=""
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743656    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743659    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743662    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743665    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743668    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743671    2560 flags.go:64] FLAG: --help="false"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743674    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-131-146.ec2.internal"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743678    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743680    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743684    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743688    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743691    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743694    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743697    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743700    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743703    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743705    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743709    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743712    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743715    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743718    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 14:05:02.750225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743721    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743724    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743727    2560 flags.go:64] FLAG: --lock-file=""
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743729    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743732    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743736    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743741    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743745    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743748    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743752    2560 flags.go:64] FLAG: --logging-format="text"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743756    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743760    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743763    2560 flags.go:64] FLAG: --manifest-url=""
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743766    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743770    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743773    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743777    2560 flags.go:64] FLAG: --max-pods="110"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743780    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743783    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743787    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743790    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743793    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743796    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743799    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743807    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 14:05:02.750839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743810    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743813    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743816    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743819    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743825    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743828    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743831    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743834    2560 flags.go:64] FLAG: --port="10250"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743837    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743841    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f3534ef0894c06ee"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743844    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743847    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743850    2560 flags.go:64] FLAG: --register-node="true"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743853    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743857    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743861    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743866    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743869    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743874    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743878    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743881    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743884    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743887    2560 flags.go:64] FLAG: --runonce="false"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743890    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743893    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 14:05:02.751486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743896    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743899    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743902    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743905    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743908    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743912    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743915    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743918    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743920    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743923    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743926    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743929    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743932    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743937    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743940    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743943    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743947    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743950    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743953    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743956    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743959    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743962    2560 flags.go:64] FLAG: --v="2"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743966    2560 flags.go:64] FLAG: --version="false"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743972    2560 flags.go:64] FLAG: --vmodule=""
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743976    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 14:05:02.752100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.743981    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744111    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744116    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744119    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744122    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744125    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744128    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744133    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744136    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744139    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744143    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744146    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744149    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744153    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744155    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744158    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744161    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744173    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744177    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744179    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:05:02.752719 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744182    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744184    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744187    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744190    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744193    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744197    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744201    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744204    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744207    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744210    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744214    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744218    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744223    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744226    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744229    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744232    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744235    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744237    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744241    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:05:02.753204 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744243    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744246    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744249    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744252    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744255    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744257    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744260    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744263    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744266    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744269    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744271    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744274    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744277    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744279    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744282    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744285    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744288    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744291    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744294    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744296    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744299    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:05:02.753718 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744302    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744304    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744308    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744311    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744315    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744318    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744321    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744323    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744326    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744328    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744331    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744334    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744337    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744339    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744342    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744345    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744347    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744350    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744353    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744367    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:05:02.754219 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744369    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744372    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744375    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744377    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744380    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744384    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744387    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744390    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744393    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744395    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744398    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744401    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744403    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744406    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744409    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744413    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744416    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744419    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744422    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:05:02.754749 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.744425    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.745147    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.753311    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.753329    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.754331    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.754341    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754417    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754423    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754427    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754430    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754434    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754437    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754440    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754443    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754446    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:05:02.755226 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754450    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754453    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754455    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754458    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754461    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754464    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754467    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754470    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754473    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754476    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754478    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754481    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754484    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754486    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754489    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754492    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754495    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754498    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754502    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754506    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:05:02.755651 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754511    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754514    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754517    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754520    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754522    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754527    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754532    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754535    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754538    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754542    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754544    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754547    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754550    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754553    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754555    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754558    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754561    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754564    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754567    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:05:02.756128 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754570    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754573    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754575    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754578    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754581    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754584    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754586    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754589    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754592    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754595    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754598    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754601    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754604    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754608    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754611    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754614    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754617    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754619    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754622    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754625    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754628    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:05:02.756606 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754630    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754633    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754636    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754639    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754641    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754644    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754647    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754649    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754652    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754655    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754657    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754660    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754663    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754665    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754668    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754671    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754674    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754676    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754679    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:05:02.757152 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754682    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754684    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754687    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754689    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754692    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754696    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754698    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754702    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754705    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754708    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754710    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.754716    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754839    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754845    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754849    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:05:02.757631 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754852    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754855    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754858    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754861    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754865    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754869    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754872    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754875    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754877    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754880    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754883    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754886    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754888    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754891    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754894    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754897    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754900    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754903    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754905    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754908    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:05:02.758007 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754911    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754914    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754917    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754920    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754923    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754926    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754931    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754934    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754937    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754940    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754942    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754945    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754948    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754950    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754953    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754956    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754959    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754961    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754964    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754967    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:05:02.758520 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754969    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754972    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754975    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754978    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754981    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754983    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754986    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754989    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754991    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754994    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754997    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.754999    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755002    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755005    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755008    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755011    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755014    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755017    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755020    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755023    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:05:02.758997 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755026    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755029    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755032    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755035    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755038    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755040    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755043    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755045    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755048    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755051    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755054    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755057    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755059    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755062    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755066    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755068    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755071    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755074    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755076    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:05:02.759684 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755079    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755082    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755085    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755087    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755090    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755093    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755096    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755099    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755101    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755104    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755107    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755109    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755113    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755117    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755120    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755124    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:05:02.760141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:02.755127    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:05:02.760537 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.755132    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 14:05:02.760537 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.755270    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 14:05:02.760537 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.757936    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 14:05:02.760537 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.759014    2560 server.go:1019] "Starting client certificate rotation"
Sep 17 14:05:02.760537 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.759111    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 14:05:02.760537 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.760146    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 14:05:02.762115 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.762093    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 14:05:02.763904 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.763873    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 14:05:02.776024 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.775998    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 17 14:05:02.779913 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.779895    2560 log.go:25] "Validated CRI v1 image API"
Sep 17 14:05:02.780677 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.780663    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 14:05:02.783583 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.783562    2560 fs.go:135] Filesystem UUIDs: map[1070de2b-17e4-4204-a4ed-36c5067eb20a:/dev/nvme0n1p4 40aee1fa-2ca7-458f-a170-47578cc96bd7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 14:05:02.783657 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.783582    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 14:05:02.789629 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.789495    2560 manager.go:216] Machine: {Timestamp:2026-09-17 14:05:02.788016632 +0000 UTC m=+0.379192390 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098840 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24332e595fec2fa9fcf889debf7c9a SystemUUID:ec24332e-595f-ec2f-a9fc-f889debf7c9a BootID:4b2c88b6-45bf-4ef9-9358-c2e242be2f9d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f1:f4:7f:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f1:f4:7f:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:92:ea:f1:be:a1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 14:05:02.789629 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.789621    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 14:05:02.789789 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.789725    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 14:05:02.792594 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.792558    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 14:05:02.792780 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.792598    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-146.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 14:05:02.793302 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.793287    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 14:05:02.794438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.794424    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 14:05:02.794496 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.794443    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 14:05:02.794496 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.794452    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 14:05:02.795212 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.795201    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 14:05:02.796417 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.796406    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 14:05:02.800734 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.800718    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 14:05:02.800864 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.800854    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 14:05:02.800915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.800883    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 14:05:02.800915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.800894    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 14:05:02.800915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.800909    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 14:05:02.801028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.800920    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 17 14:05:02.801028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.800931    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 14:05:02.801372 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.801348    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 14:05:02.801440 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.801382    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 14:05:02.803726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.803703    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 14:05:02.806441 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.806427    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 14:05:02.806507 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.806457    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 14:05:02.808558 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808545    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 14:05:02.808610 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808562    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 14:05:02.808610 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808574    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 14:05:02.808610 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808580    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 14:05:02.808610 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808589    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 14:05:02.808610 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808597    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 14:05:02.808742 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808613    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 14:05:02.808742 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808619    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 14:05:02.808742 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808626    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 14:05:02.808742 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808632    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 14:05:02.808742 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808645    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 14:05:02.808742 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.808651    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 14:05:02.809439 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.809427    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 14:05:02.809439 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.809440    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 14:05:02.813207 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.813185    2560 server.go:1268] "Started kubelet"
Sep 17 14:05:02.813340 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.813260    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 14:05:02.813411 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.813322    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 14:05:02.813455 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.813448    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 14:05:02.814404 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.814381    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-146.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 14:05:02.814438 ip-10-0-131-146 systemd[1]: Started Kubernetes Kubelet.
Sep 17 14:05:02.814689 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.814516    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 14:05:02.817011 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.816967    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 14:05:02.817097 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.817012    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 14:05:02.817148 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.817104    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 14:05:02.820240 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.820218    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 14:05:02.822496 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.821367    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-146.ec2.internal.18d620be1c11c016  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-146.ec2.internal,UID:ip-10-0-131-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-146.ec2.internal,},FirstTimestamp:2026-09-17 14:05:02.813151254 +0000 UTC m=+0.404327007,LastTimestamp:2026-09-17 14:05:02.813151254 +0000 UTC m=+0.404327007,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-146.ec2.internal,}"
Sep 17 14:05:02.827015 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.826988    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 14:05:02.827499 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.827482    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 14:05:02.827830 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.827809    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:02.828602 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828585    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 14:05:02.828676 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828608    2560 factory.go:153] Registering CRI-O factory
Sep 17 14:05:02.828676 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828587    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 14:05:02.828676 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828627    2560 factory.go:223] Registration of the crio container factory successfully
Sep 17 14:05:02.828676 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828635    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 14:05:02.828896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828711    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 14:05:02.828896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828722    2560 factory.go:55] Registering systemd factory
Sep 17 14:05:02.828896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828731    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 17 14:05:02.828896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828749    2560 factory.go:103] Registering Raw factory
Sep 17 14:05:02.828896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828763    2560 manager.go:1196] Started watching for new ooms in manager
Sep 17 14:05:02.828896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828795    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 14:05:02.828896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.828803    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 14:05:02.829201 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.829165    2560 manager.go:349] Starting recovery of all containers
Sep 17 14:05:02.829941 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.829914    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 14:05:02.830137 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.830110    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 14:05:02.840099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.839950    2560 manager.go:354] Recovery completed
Sep 17 14:05:02.844244 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.844227    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:05:02.846405 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.846382    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pwcg2"
Sep 17 14:05:02.846405 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.846392    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:05:02.846543 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.846439    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:05:02.846543 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.846451    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:05:02.847081 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.847058    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 14:05:02.847081 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.847081    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 14:05:02.847208 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.847103    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 14:05:02.847208 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.847061    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:05:02.847856 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.847834    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:05:02.847915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.847873    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:05:02.847915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.847889    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:05:02.849384 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.849351    2560 policy_none.go:49] "None policy: Start"
Sep 17 14:05:02.849384 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.849384    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 14:05:02.849483 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.849398    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 14:05:02.851349 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.851336    2560 policy_none.go:47] "Start"
Sep 17 14:05:02.856475 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.856315    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pwcg2"
Sep 17 14:05:02.857527 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.857441    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-146.ec2.internal.18d620be1e0d3c8a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-146.ec2.internal,UID:ip-10-0-131-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-146.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-146.ec2.internal,},FirstTimestamp:2026-09-17 14:05:02.846409866 +0000 UTC m=+0.437585618,LastTimestamp:2026-09-17 14:05:02.846409866 +0000 UTC m=+0.437585618,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-146.ec2.internal,}"
Sep 17 14:05:02.878097 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.878075    2560 manager.go:337] "Starting Device Plugin manager"
Sep 17 14:05:02.878249 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.878128    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 14:05:02.878249 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.878141    2560 server.go:87] "Starting device plugin registration server"
Sep 17 14:05:02.878588 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.878575    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 14:05:02.878641 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.878591    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 14:05:02.878739 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.878724    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 14:05:02.878823 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.878809    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 14:05:02.878823 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.878819    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 14:05:02.879447 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.879427    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 14:05:02.879533 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.879472    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:02.957492 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.957453    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 14:05:02.959412 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.958802    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 14:05:02.959412 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.958862    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 14:05:02.959412 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.958890    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 14:05:02.959412 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.958981    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 14:05:02.961350 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.961325    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:05:02.979205 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.979171    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:05:02.980400 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.980382    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:05:02.980482 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.980412    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:05:02.980482 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.980424    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:05:02.980482 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.980451    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-146.ec2.internal"
Sep 17 14:05:02.990123 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:02.990044    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-146.ec2.internal"
Sep 17 14:05:02.990123 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:02.990074    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-146.ec2.internal\": node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:03.043667 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.043624    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:03.059890 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.059855    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal"]
Sep 17 14:05:03.060027 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.059956    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:05:03.061775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.061758    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:05:03.061856 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.061789    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:05:03.061856 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.061800    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:05:03.063764 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.063749    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:05:03.063891 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.063876    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.063948 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.063913    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:05:03.064560 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.064540    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:05:03.064656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.064568    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:05:03.064656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.064578    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:05:03.064656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.064542    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:05:03.064656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.064646    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:05:03.064656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.064657    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:05:03.066633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.066618    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.066710 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.066644    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:05:03.067314 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.067299    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:05:03.067407 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.067324    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:05:03.067407 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.067337    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:05:03.088252 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.088220    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-146.ec2.internal\" not found" node="ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.092831 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.092812    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-146.ec2.internal\" not found" node="ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.130691 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.130654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2a8032d1e1ba5143915f9592a048d25-config\") pod \"kube-apiserver-proxy-ip-10-0-131-146.ec2.internal\" (UID: \"f2a8032d1e1ba5143915f9592a048d25\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.130691 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.130693    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e598cbcd04df48c89405a63ef0a1f2f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal\" (UID: \"e598cbcd04df48c89405a63ef0a1f2f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.130893 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.130717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e598cbcd04df48c89405a63ef0a1f2f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal\" (UID: \"e598cbcd04df48c89405a63ef0a1f2f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.143882 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.143847    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:03.231720 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.231682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e598cbcd04df48c89405a63ef0a1f2f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal\" (UID: \"e598cbcd04df48c89405a63ef0a1f2f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.231720 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.231722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2a8032d1e1ba5143915f9592a048d25-config\") pod \"kube-apiserver-proxy-ip-10-0-131-146.ec2.internal\" (UID: \"f2a8032d1e1ba5143915f9592a048d25\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.231902 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.231737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e598cbcd04df48c89405a63ef0a1f2f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal\" (UID: \"e598cbcd04df48c89405a63ef0a1f2f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.231902 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.231781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e598cbcd04df48c89405a63ef0a1f2f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal\" (UID: \"e598cbcd04df48c89405a63ef0a1f2f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.231902 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.231784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e598cbcd04df48c89405a63ef0a1f2f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal\" (UID: \"e598cbcd04df48c89405a63ef0a1f2f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.231902 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.231802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2a8032d1e1ba5143915f9592a048d25-config\") pod \"kube-apiserver-proxy-ip-10-0-131-146.ec2.internal\" (UID: \"f2a8032d1e1ba5143915f9592a048d25\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.244864 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.244800    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:03.345728 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.345685    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:03.392882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.392857    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.396400 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.396380    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.446516 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.446480    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:03.547031 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.546963    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:03.647582 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.647539    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:03.748088 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.748052    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:03.759268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.759228    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 14:05:03.759471 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.759432    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 14:05:03.832317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.832288    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:05:03.851663 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:03.849465    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-146.ec2.internal\" not found"
Sep 17 14:05:03.860325 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.860282    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 14:00:03 +0000 UTC" deadline="2028-04-30 06:42:51.05475618 +0000 UTC"
Sep 17 14:05:03.860325 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.860317    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14176h37m47.19444264s"
Sep 17 14:05:03.866561 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.866540    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:05:03.928060 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.928027    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.939813 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.939789    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 14:05:03.940751 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.940735    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal"
Sep 17 14:05:03.947049 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.947028    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:05:03.949573 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.949555    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 14:05:03.959176 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:03.959141    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode598cbcd04df48c89405a63ef0a1f2f5.slice/crio-a57502a0ba14fc90e12bab28440dde421262876b678b9e504a393eb02cf5252f WatchSource:0}: Error finding container a57502a0ba14fc90e12bab28440dde421262876b678b9e504a393eb02cf5252f: Status 404 returned error can't find the container with id a57502a0ba14fc90e12bab28440dde421262876b678b9e504a393eb02cf5252f
Sep 17 14:05:03.965271 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:03.965257    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:05:03.990272 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:03.990247    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a8032d1e1ba5143915f9592a048d25.slice/crio-1cd561ca7e0601653dee98d75b7c1335ebf5bf6beb8174ebcadba337a8919a07 WatchSource:0}: Error finding container 1cd561ca7e0601653dee98d75b7c1335ebf5bf6beb8174ebcadba337a8919a07: Status 404 returned error can't find the container with id 1cd561ca7e0601653dee98d75b7c1335ebf5bf6beb8174ebcadba337a8919a07
Sep 17 14:05:04.640932 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.640900    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:05:04.801973 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.801938    2560 apiserver.go:52] "Watching apiserver"
Sep 17 14:05:04.810696 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.810667    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 14:05:04.811078 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.811053    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5","openshift-cluster-node-tuning-operator/tuned-fnq7p","openshift-dns/node-resolver-gc5s9","openshift-image-registry/node-ca-h2l92","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal","kube-system/global-pull-secret-syncer-52kgw","openshift-multus/multus-additional-cni-plugins-7cns4","openshift-multus/multus-wjnk2","openshift-multus/network-metrics-daemon-m4bth","openshift-network-diagnostics/network-check-target-lpgsd","openshift-network-operator/iptables-alerter-2lrfm","openshift-ovn-kubernetes/ovnkube-node-kpjkj","kube-system/konnectivity-agent-wr9bq","kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal"]
Sep 17 14:05:04.813743 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.813721    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.815838 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.815809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:04.816582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.816496    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 14:05:04.816582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.816497    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 14:05:04.816744 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.816583    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 14:05:04.816744 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.816636    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 14:05:04.816881 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.816864    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wjt26\""
Sep 17 14:05:04.816967 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.816950    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 14:05:04.818047 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.818019    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:04.818157 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.818133    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:05:04.818549 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.818530    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 14:05:04.818549 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.818539    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7xcmx\""
Sep 17 14:05:04.818751 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.818605    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 14:05:04.820263 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.820242    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:04.820416 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.820390    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 14:05:04.820491 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.820432    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-587tq\""
Sep 17 14:05:04.820624 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.820604    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 14:05:04.823428 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.822486    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bjrvl\""
Sep 17 14:05:04.823428 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.822708    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 14:05:04.823428 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.822886    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 14:05:04.823428 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.822932    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 14:05:04.823428 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.822960    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.824828 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.824809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:04.824925 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:04.824890    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:04.825237 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.825222    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 14:05:04.825719 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.825699    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5zps9\""
Sep 17 14:05:04.827197 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.827173    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:04.827292 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:04.827247    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:04.827899 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.827880    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 14:05:04.829573 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.829555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:04.832206 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.832186    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 14:05:04.832325 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.832307    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 14:05:04.832681 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.832666    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4x77w\""
Sep 17 14:05:04.833382 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.833340    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.835690 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.835668    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:04.835808 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:04.835750    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:04.836348 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.836324    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ndpsd\""
Sep 17 14:05:04.836449 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.836370    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 14:05:04.836582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.836564    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:05:04.838294 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.838270    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:04.840540 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmg9p\" (UniqueName: \"kubernetes.io/projected/d18eac27-3f6f-4757-94c9-0e01cd87d96a-kube-api-access-xmg9p\") pod \"node-ca-h2l92\" (UID: \"d18eac27-3f6f-4757-94c9-0e01cd87d96a\") " pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:04.840632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3839cd65-b24b-4e79-b075-d703429c694f-kubelet-config\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:04.840632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-var-lib-cni-bin\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.840632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840595    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-52pb5\""
Sep 17 14:05:04.840632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840602    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 14:05:04.840769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840608    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-var-lib-cni-multus\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.840769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-hostroot\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.840769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-run-multus-certs\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.840874 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840770    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.840874 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840799    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-cni-dir\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.840874 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840818    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 14:05:04.840874 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840822    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-var-lib-kubelet\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqccm\" (UniqueName: \"kubernetes.io/projected/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-kube-api-access-nqccm\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840982    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c5ef9a9-f846-49ad-885b-dba324eea698-iptables-alerter-script\") pod \"iptables-alerter-2lrfm\" (UID: \"9c5ef9a9-f846-49ad-885b-dba324eea698\") " pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:04.841095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.840988    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 14:05:04.841095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841007    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7975\" (UniqueName: \"kubernetes.io/projected/9c5ef9a9-f846-49ad-885b-dba324eea698-kube-api-access-t7975\") pod \"iptables-alerter-2lrfm\" (UID: \"9c5ef9a9-f846-49ad-885b-dba324eea698\") " pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:04.841095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-run-k8s-cni-cncf-io\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:04.841095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29b2af19-f2bf-42d3-9716-639cb0848d7b-konnectivity-ca\") pod \"konnectivity-agent-wr9bq\" (UID: \"29b2af19-f2bf-42d3-9716-639cb0848d7b\") " pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:04.841438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.841438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841185    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26394392-22b2-42a8-98b8-a6b6f194f569-tmp-dir\") pod \"node-resolver-gc5s9\" (UID: \"26394392-22b2-42a8-98b8-a6b6f194f569\") " pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:04.841438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d18eac27-3f6f-4757-94c9-0e01cd87d96a-host\") pod \"node-ca-h2l92\" (UID: \"d18eac27-3f6f-4757-94c9-0e01cd87d96a\") " pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:04.841438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.841438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-cnibin\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-cni-binary-copy\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841299    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3839cd65-b24b-4e79-b075-d703429c694f-dbus\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:04.841438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-cnibin\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.841438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841378    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-run-netns\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841416    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-conf-dir\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvvx\" (UniqueName: \"kubernetes.io/projected/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-kube-api-access-ssvvx\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26394392-22b2-42a8-98b8-a6b6f194f569-hosts-file\") pod \"node-resolver-gc5s9\" (UID: \"26394392-22b2-42a8-98b8-a6b6f194f569\") " pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tdpt\" (UniqueName: \"kubernetes.io/projected/26394392-22b2-42a8-98b8-a6b6f194f569-kube-api-access-6tdpt\") pod \"node-resolver-gc5s9\" (UID: \"26394392-22b2-42a8-98b8-a6b6f194f569\") " pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841542    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d18eac27-3f6f-4757-94c9-0e01cd87d96a-serviceca\") pod \"node-ca-h2l92\" (UID: \"d18eac27-3f6f-4757-94c9-0e01cd87d96a\") " pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-system-cni-dir\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-socket-dir-parent\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-daemon-config\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-system-cni-dir\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841651    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-etc-kubernetes\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841676    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c5ef9a9-f846-49ad-885b-dba324eea698-host-slash\") pod \"iptables-alerter-2lrfm\" (UID: \"9c5ef9a9-f846-49ad-885b-dba324eea698\") " pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841691    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841706    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29b2af19-f2bf-42d3-9716-639cb0848d7b-agent-certs\") pod \"konnectivity-agent-wr9bq\" (UID: \"29b2af19-f2bf-42d3-9716-639cb0848d7b\") " pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:04.841882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.841721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f677h\" (UniqueName: \"kubernetes.io/projected/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-kube-api-access-f677h\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.845477 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.845444    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 14:05:04.845715 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.845692    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 14:05:04.845889 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.845874    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 14:05:04.846172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.846158    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 14:05:04.847679 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.847645    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 14:05:04.848276 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.848251    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9qxwm\""
Sep 17 14:05:04.848641 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.848623    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 14:05:04.848729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.848669    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 14:05:04.898040 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.897978    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ttcfv"
Sep 17 14:05:04.910813 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.910783    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ttcfv"
Sep 17 14:05:04.929582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.929558    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 14:05:04.942216 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:04.942216 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-lib-modules\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.942459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/644ea3fc-db18-4797-8d33-246b855769c9-tmp\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.942459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f677h\" (UniqueName: \"kubernetes.io/projected/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-kube-api-access-f677h\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.942459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmg9p\" (UniqueName: \"kubernetes.io/projected/d18eac27-3f6f-4757-94c9-0e01cd87d96a-kube-api-access-xmg9p\") pod \"node-ca-h2l92\" (UID: \"d18eac27-3f6f-4757-94c9-0e01cd87d96a\") " pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:04.942459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3839cd65-b24b-4e79-b075-d703429c694f-kubelet-config\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:04.942459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942398    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-run-systemd\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.942459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ae503a2-b685-4d50-96be-ae40f573337d-ovn-node-metrics-cert\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.942459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ae503a2-b685-4d50-96be-ae40f573337d-ovnkube-script-lib\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.942459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942454    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-run\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942482    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-host\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3839cd65-b24b-4e79-b075-d703429c694f-kubelet-config\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-var-lib-cni-bin\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-hostroot\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-run-multus-certs\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-var-lib-cni-bin\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psr26\" (UniqueName: \"kubernetes.io/projected/644ea3fc-db18-4797-8d33-246b855769c9-kube-api-access-psr26\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942662    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-run-multus-certs\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-hostroot\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqccm\" (UniqueName: \"kubernetes.io/projected/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-kube-api-access-nqccm\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c5ef9a9-f846-49ad-885b-dba324eea698-iptables-alerter-script\") pod \"iptables-alerter-2lrfm\" (UID: \"9c5ef9a9-f846-49ad-885b-dba324eea698\") " pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-cni-bin\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942768    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdkmn\" (UniqueName: \"kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn\") pod \"network-check-target-lpgsd\" (UID: \"582bd8f2-b198-4a29-96d1-a838e1dad55c\") " pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-run-k8s-cni-cncf-io\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:04.942840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29b2af19-f2bf-42d3-9716-639cb0848d7b-konnectivity-ca\") pod \"konnectivity-agent-wr9bq\" (UID: \"29b2af19-f2bf-42d3-9716-639cb0848d7b\") " pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-run-k8s-cni-cncf-io\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942873    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-cni-binary-copy\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-systemd-units\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:04.942920    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942933    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-run-netns\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.942957    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-device-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:04.943010    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs podName:796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e nodeName:}" failed. No retries permitted until 2026-09-17 14:05:05.442965408 +0000 UTC m=+3.034141148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs") pod "network-metrics-daemon-m4bth" (UID: "796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943077    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-sysctl-d\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-cnibin\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-run-netns\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-conf-dir\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943193    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ae503a2-b685-4d50-96be-ae40f573337d-ovnkube-config\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943201    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-cnibin\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-kubernetes\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26394392-22b2-42a8-98b8-a6b6f194f569-hosts-file\") pod \"node-resolver-gc5s9\" (UID: \"26394392-22b2-42a8-98b8-a6b6f194f569\") " pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d18eac27-3f6f-4757-94c9-0e01cd87d96a-serviceca\") pod \"node-ca-h2l92\" (UID: \"d18eac27-3f6f-4757-94c9-0e01cd87d96a\") " pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:04.943604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-run-netns\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29b2af19-f2bf-42d3-9716-639cb0848d7b-agent-certs\") pod \"konnectivity-agent-wr9bq\" (UID: \"29b2af19-f2bf-42d3-9716-639cb0848d7b\") " pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26394392-22b2-42a8-98b8-a6b6f194f569-hosts-file\") pod \"node-resolver-gc5s9\" (UID: \"26394392-22b2-42a8-98b8-a6b6f194f569\") " pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943340    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-run-openvswitch\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943377    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-systemd\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943400    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/644ea3fc-db18-4797-8d33-246b855769c9-etc-tuned\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-system-cni-dir\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-conf-dir\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c5ef9a9-f846-49ad-885b-dba324eea698-iptables-alerter-script\") pod \"iptables-alerter-2lrfm\" (UID: \"9c5ef9a9-f846-49ad-885b-dba324eea698\") " pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943485    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-system-cni-dir\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943486    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:04.943567    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29b2af19-f2bf-42d3-9716-639cb0848d7b-konnectivity-ca\") pod \"konnectivity-agent-wr9bq\" (UID: \"29b2af19-f2bf-42d3-9716-639cb0848d7b\") " pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943604    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-etc-kubernetes\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:04.943620    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret podName:3839cd65-b24b-4e79-b075-d703429c694f nodeName:}" failed. No retries permitted until 2026-09-17 14:05:05.443603306 +0000 UTC m=+3.034779046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret") pod "global-pull-secret-syncer-52kgw" (UID: "3839cd65-b24b-4e79-b075-d703429c694f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c5ef9a9-f846-49ad-885b-dba324eea698-host-slash\") pod \"iptables-alerter-2lrfm\" (UID: \"9c5ef9a9-f846-49ad-885b-dba324eea698\") " pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:04.944501 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-etc-kubernetes\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-etc-selinux\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c5ef9a9-f846-49ad-885b-dba324eea698-host-slash\") pod \"iptables-alerter-2lrfm\" (UID: \"9c5ef9a9-f846-49ad-885b-dba324eea698\") " pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-sys-fs\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943740    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-node-log\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d18eac27-3f6f-4757-94c9-0e01cd87d96a-serviceca\") pod \"node-ca-h2l92\" (UID: \"d18eac27-3f6f-4757-94c9-0e01cd87d96a\") " pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-var-lib-cni-multus\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-log-socket\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-cni-dir\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-var-lib-cni-multus\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-var-lib-kubelet\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7975\" (UniqueName: \"kubernetes.io/projected/9c5ef9a9-f846-49ad-885b-dba324eea698-kube-api-access-t7975\") pod \"iptables-alerter-2lrfm\" (UID: \"9c5ef9a9-f846-49ad-885b-dba324eea698\") " pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-host-var-lib-kubelet\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-run-ovn\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnlsp\" (UniqueName: \"kubernetes.io/projected/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-kube-api-access-hnlsp\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:04.945317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.943994    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-cni-dir\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-sysconfig\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26394392-22b2-42a8-98b8-a6b6f194f569-tmp-dir\") pod \"node-resolver-gc5s9\" (UID: \"26394392-22b2-42a8-98b8-a6b6f194f569\") " pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d18eac27-3f6f-4757-94c9-0e01cd87d96a-host\") pod \"node-ca-h2l92\" (UID: \"d18eac27-3f6f-4757-94c9-0e01cd87d96a\") " pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-cnibin\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3839cd65-b24b-4e79-b075-d703429c694f-dbus\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944194    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d18eac27-3f6f-4757-94c9-0e01cd87d96a-host\") pod \"node-ca-h2l92\" (UID: \"d18eac27-3f6f-4757-94c9-0e01cd87d96a\") " pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-etc-openvswitch\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944226    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-cni-netd\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-cnibin\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944255    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-registration-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvvx\" (UniqueName: \"kubernetes.io/projected/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-kube-api-access-ssvvx\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3839cd65-b24b-4e79-b075-d703429c694f-dbus\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-var-lib-openvswitch\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.946210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-socket-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944427    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26394392-22b2-42a8-98b8-a6b6f194f569-tmp-dir\") pod \"node-resolver-gc5s9\" (UID: \"26394392-22b2-42a8-98b8-a6b6f194f569\") " pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944449    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-modprobe-d\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944479    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-cni-binary-copy\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-sysctl-conf\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944531    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-sys\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-var-lib-kubelet\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944594    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tdpt\" (UniqueName: \"kubernetes.io/projected/26394392-22b2-42a8-98b8-a6b6f194f569-kube-api-access-6tdpt\") pod \"node-resolver-gc5s9\" (UID: \"26394392-22b2-42a8-98b8-a6b6f194f569\") " pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-system-cni-dir\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-socket-dir-parent\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-daemon-config\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944758    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-slash\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944787    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-system-cni-dir\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.947025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944806    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-kubelet\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.947877 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ae503a2-b685-4d50-96be-ae40f573337d-env-overrides\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.947877 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-socket-dir-parent\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.947877 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.944884    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm5hx\" (UniqueName: \"kubernetes.io/projected/9ae503a2-b685-4d50-96be-ae40f573337d-kube-api-access-nm5hx\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:04.947877 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.945130    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.947877 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.945232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-multus-daemon-config\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.948147 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.948070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29b2af19-f2bf-42d3-9716-639cb0848d7b-agent-certs\") pod \"konnectivity-agent-wr9bq\" (UID: \"29b2af19-f2bf-42d3-9716-639cb0848d7b\") " pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:04.952275 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.952203    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqccm\" (UniqueName: \"kubernetes.io/projected/9aabfb62-00a4-4dca-8796-ee6bcd6a1277-kube-api-access-nqccm\") pod \"multus-wjnk2\" (UID: \"9aabfb62-00a4-4dca-8796-ee6bcd6a1277\") " pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:04.952589 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.952566    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmg9p\" (UniqueName: \"kubernetes.io/projected/d18eac27-3f6f-4757-94c9-0e01cd87d96a-kube-api-access-xmg9p\") pod \"node-ca-h2l92\" (UID: \"d18eac27-3f6f-4757-94c9-0e01cd87d96a\") " pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:04.952965 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.952941    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f677h\" (UniqueName: \"kubernetes.io/projected/c1c3e777-dd90-4fea-9735-157ef5f2ebb7-kube-api-access-f677h\") pod \"multus-additional-cni-plugins-7cns4\" (UID: \"c1c3e777-dd90-4fea-9735-157ef5f2ebb7\") " pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:04.955368 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.955327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7975\" (UniqueName: \"kubernetes.io/projected/9c5ef9a9-f846-49ad-885b-dba324eea698-kube-api-access-t7975\") pod \"iptables-alerter-2lrfm\" (UID: \"9c5ef9a9-f846-49ad-885b-dba324eea698\") " pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:04.956014 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.955997    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvvx\" (UniqueName: \"kubernetes.io/projected/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-kube-api-access-ssvvx\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:04.956180 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.956152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tdpt\" (UniqueName: \"kubernetes.io/projected/26394392-22b2-42a8-98b8-a6b6f194f569-kube-api-access-6tdpt\") pod \"node-resolver-gc5s9\" (UID: \"26394392-22b2-42a8-98b8-a6b6f194f569\") " pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:04.963834 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.963794    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal" event={"ID":"f2a8032d1e1ba5143915f9592a048d25","Type":"ContainerStarted","Data":"1cd561ca7e0601653dee98d75b7c1335ebf5bf6beb8174ebcadba337a8919a07"}
Sep 17 14:05:04.964936 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:04.964914    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal" event={"ID":"e598cbcd04df48c89405a63ef0a1f2f5","Type":"ContainerStarted","Data":"a57502a0ba14fc90e12bab28440dde421262876b678b9e504a393eb02cf5252f"}
Sep 17 14:05:05.045992 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.045944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-systemd-units\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.045992 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.045996    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-run-netns\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-device-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.046257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046035    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-systemd-units\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-sysctl-d\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.046257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046094    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-run-netns\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-device-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.046257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046137    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ae503a2-b685-4d50-96be-ae40f573337d-ovnkube-config\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-kubernetes\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.046257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-sysctl-d\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.046257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-run-openvswitch\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-kubernetes\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-systemd\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046278    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-run-openvswitch\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/644ea3fc-db18-4797-8d33-246b855769c9-etc-tuned\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046318    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-systemd\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-etc-selinux\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-sys-fs\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046418    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-node-log\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-log-socket\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046467    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-run-ovn\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlsp\" (UniqueName: \"kubernetes.io/projected/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-kube-api-access-hnlsp\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-etc-selinux\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-sysconfig\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-log-socket\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046554    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-etc-openvswitch\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-cni-netd\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046602    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-sys-fs\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.046775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-registration-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-var-lib-openvswitch\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046656    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-registration-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046670    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-run-ovn\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-socket-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046729    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-modprobe-d\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-sysctl-conf\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-sys\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046783    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-sysconfig\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-var-lib-kubelet\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046808    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-socket-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ae503a2-b685-4d50-96be-ae40f573337d-ovnkube-config\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046853    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046857    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-var-lib-openvswitch\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-etc-openvswitch\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046899    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-var-lib-kubelet\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.047656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-cni-netd\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.046942    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-sys\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-modprobe-d\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047022    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-etc-sysctl-conf\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-node-log\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-slash\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047147    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-kubelet\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-slash\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047168    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ae503a2-b685-4d50-96be-ae40f573337d-env-overrides\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-kubelet\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047229    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm5hx\" (UniqueName: \"kubernetes.io/projected/9ae503a2-b685-4d50-96be-ae40f573337d-kube-api-access-nm5hx\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047277    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-lib-modules\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/644ea3fc-db18-4797-8d33-246b855769c9-tmp\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-run-systemd\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.048532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ae503a2-b685-4d50-96be-ae40f573337d-ovn-node-metrics-cert\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ae503a2-b685-4d50-96be-ae40f573337d-ovnkube-script-lib\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047352    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-run\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047410    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-lib-modules\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047464    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-run\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ae503a2-b685-4d50-96be-ae40f573337d-env-overrides\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-host\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-run-systemd\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/644ea3fc-db18-4797-8d33-246b855769c9-host\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psr26\" (UniqueName: \"kubernetes.io/projected/644ea3fc-db18-4797-8d33-246b855769c9-kube-api-access-psr26\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-cni-bin\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkmn\" (UniqueName: \"kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn\") pod \"network-check-target-lpgsd\" (UID: \"582bd8f2-b198-4a29-96d1-a838e1dad55c\") " pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ae503a2-b685-4d50-96be-ae40f573337d-host-cni-bin\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.049726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.047946    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ae503a2-b685-4d50-96be-ae40f573337d-ovnkube-script-lib\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.050305 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.049734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/644ea3fc-db18-4797-8d33-246b855769c9-etc-tuned\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.050305 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.049990    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/644ea3fc-db18-4797-8d33-246b855769c9-tmp\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.050305 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.050121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ae503a2-b685-4d50-96be-ae40f573337d-ovn-node-metrics-cert\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.056005 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.055676    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:05:05.056005 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.055701    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:05:05.056005 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.055716    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdkmn for pod openshift-network-diagnostics/network-check-target-lpgsd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:05.056005 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.055802    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn podName:582bd8f2-b198-4a29-96d1-a838e1dad55c nodeName:}" failed. No retries permitted until 2026-09-17 14:05:05.555782666 +0000 UTC m=+3.146958435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gdkmn" (UniqueName: "kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn") pod "network-check-target-lpgsd" (UID: "582bd8f2-b198-4a29-96d1-a838e1dad55c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:05.058213 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.058184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlsp\" (UniqueName: \"kubernetes.io/projected/1672da8f-e4d5-4ca6-94d4-f9256ef75bda-kube-api-access-hnlsp\") pod \"aws-ebs-csi-driver-node-j5qf5\" (UID: \"1672da8f-e4d5-4ca6-94d4-f9256ef75bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.058345 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.058326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psr26\" (UniqueName: \"kubernetes.io/projected/644ea3fc-db18-4797-8d33-246b855769c9-kube-api-access-psr26\") pod \"tuned-fnq7p\" (UID: \"644ea3fc-db18-4797-8d33-246b855769c9\") " pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.058459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.058439    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm5hx\" (UniqueName: \"kubernetes.io/projected/9ae503a2-b685-4d50-96be-ae40f573337d-kube-api-access-nm5hx\") pod \"ovnkube-node-kpjkj\" (UID: \"9ae503a2-b685-4d50-96be-ae40f573337d\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.126095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.126060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7cns4"
Sep 17 14:05:05.133166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.133134    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2lrfm"
Sep 17 14:05:05.143209 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.143184    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gc5s9"
Sep 17 14:05:05.148604 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.148550    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h2l92"
Sep 17 14:05:05.156570 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.156543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wjnk2"
Sep 17 14:05:05.162525 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.162497    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:05.169977 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.169953    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fnq7p"
Sep 17 14:05:05.176780 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.176760    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5"
Sep 17 14:05:05.183724 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.183699    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:05.451154 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.451067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:05.451154 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.451116    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:05.451416 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.451255    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:05.451416 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.451347    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs podName:796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e nodeName:}" failed. No retries permitted until 2026-09-17 14:05:06.451323447 +0000 UTC m=+4.042499209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs") pod "network-metrics-daemon-m4bth" (UID: "796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:05.451416 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.451255    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:05.451584 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.451431    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret podName:3839cd65-b24b-4e79-b075-d703429c694f nodeName:}" failed. No retries permitted until 2026-09-17 14:05:06.451413812 +0000 UTC m=+4.042589574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret") pod "global-pull-secret-syncer-52kgw" (UID: "3839cd65-b24b-4e79-b075-d703429c694f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:05.643667 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:05.643639    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae503a2_b685_4d50_96be_ae40f573337d.slice/crio-705134b9e565887ba5d5d4c719658d3fdf31c63ae179e9d6d44b2550d45bb2ef WatchSource:0}: Error finding container 705134b9e565887ba5d5d4c719658d3fdf31c63ae179e9d6d44b2550d45bb2ef: Status 404 returned error can't find the container with id 705134b9e565887ba5d5d4c719658d3fdf31c63ae179e9d6d44b2550d45bb2ef
Sep 17 14:05:05.644685 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:05.644659    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1672da8f_e4d5_4ca6_94d4_f9256ef75bda.slice/crio-6bea957bce633eb9767fbc4cb591e1bf8cbc5ccc648c7441878179a510c9bc70 WatchSource:0}: Error finding container 6bea957bce633eb9767fbc4cb591e1bf8cbc5ccc648c7441878179a510c9bc70: Status 404 returned error can't find the container with id 6bea957bce633eb9767fbc4cb591e1bf8cbc5ccc648c7441878179a510c9bc70
Sep 17 14:05:05.646382 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:05.646338    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5ef9a9_f846_49ad_885b_dba324eea698.slice/crio-4bdb1436fb3f4e09e64243b30fd4ee9af961c64bc25454206e5bc319d5f671a3 WatchSource:0}: Error finding container 4bdb1436fb3f4e09e64243b30fd4ee9af961c64bc25454206e5bc319d5f671a3: Status 404 returned error can't find the container with id 4bdb1436fb3f4e09e64243b30fd4ee9af961c64bc25454206e5bc319d5f671a3
Sep 17 14:05:05.647836 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:05.647810    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aabfb62_00a4_4dca_8796_ee6bcd6a1277.slice/crio-b17b3a59a2dbebefa0828f1bb0f01ce75f81cab7e5db079f3f05b3c712ca1713 WatchSource:0}: Error finding container b17b3a59a2dbebefa0828f1bb0f01ce75f81cab7e5db079f3f05b3c712ca1713: Status 404 returned error can't find the container with id b17b3a59a2dbebefa0828f1bb0f01ce75f81cab7e5db079f3f05b3c712ca1713
Sep 17 14:05:05.649329 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:05.649307    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18eac27_3f6f_4757_94c9_0e01cd87d96a.slice/crio-5ab8d3abdff7cc50b417cd6e995e902b65101a7ac14ed2d63bc41347ecb3b872 WatchSource:0}: Error finding container 5ab8d3abdff7cc50b417cd6e995e902b65101a7ac14ed2d63bc41347ecb3b872: Status 404 returned error can't find the container with id 5ab8d3abdff7cc50b417cd6e995e902b65101a7ac14ed2d63bc41347ecb3b872
Sep 17 14:05:05.653205 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.653184    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkmn\" (UniqueName: \"kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn\") pod \"network-check-target-lpgsd\" (UID: \"582bd8f2-b198-4a29-96d1-a838e1dad55c\") " pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:05.653317 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.653305    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:05:05.653361 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.653321    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:05:05.653361 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.653331    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdkmn for pod openshift-network-diagnostics/network-check-target-lpgsd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:05.653511 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.653378    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn podName:582bd8f2-b198-4a29-96d1-a838e1dad55c nodeName:}" failed. No retries permitted until 2026-09-17 14:05:06.653363175 +0000 UTC m=+4.244538931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdkmn" (UniqueName: "kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn") pod "network-check-target-lpgsd" (UID: "582bd8f2-b198-4a29-96d1-a838e1dad55c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:05.674326 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:05.674301    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b2af19_f2bf_42d3_9716_639cb0848d7b.slice/crio-33fe9716d2b001948bdae9280c8b8580692689d721a22d7dca9f899d1a62dba5 WatchSource:0}: Error finding container 33fe9716d2b001948bdae9280c8b8580692689d721a22d7dca9f899d1a62dba5: Status 404 returned error can't find the container with id 33fe9716d2b001948bdae9280c8b8580692689d721a22d7dca9f899d1a62dba5
Sep 17 14:05:05.676235 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:05.676210    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644ea3fc_db18_4797_8d33_246b855769c9.slice/crio-d0c0015a8ac3a6d34b0040fc0400a09d5ee85a579cd5d0197014a7288d8e4088 WatchSource:0}: Error finding container d0c0015a8ac3a6d34b0040fc0400a09d5ee85a579cd5d0197014a7288d8e4088: Status 404 returned error can't find the container with id d0c0015a8ac3a6d34b0040fc0400a09d5ee85a579cd5d0197014a7288d8e4088
Sep 17 14:05:05.677480 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:05.677457    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c3e777_dd90_4fea_9735_157ef5f2ebb7.slice/crio-6fd18ce722371abe006ca2ff8e4ebb220e13ba40a521df3c602fd8bafed5efc8 WatchSource:0}: Error finding container 6fd18ce722371abe006ca2ff8e4ebb220e13ba40a521df3c602fd8bafed5efc8: Status 404 returned error can't find the container with id 6fd18ce722371abe006ca2ff8e4ebb220e13ba40a521df3c602fd8bafed5efc8
Sep 17 14:05:05.678481 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:05.678459    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26394392_22b2_42a8_98b8_a6b6f194f569.slice/crio-4ea6434ae669948aad2f56385434df41909d0bc8498e6ec8ccc9c5075fce73ef WatchSource:0}: Error finding container 4ea6434ae669948aad2f56385434df41909d0bc8498e6ec8ccc9c5075fce73ef: Status 404 returned error can't find the container with id 4ea6434ae669948aad2f56385434df41909d0bc8498e6ec8ccc9c5075fce73ef
Sep 17 14:05:05.914154 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.914100    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 14:00:05 +0000 UTC" deadline="2028-02-14 04:41:13.280569114 +0000 UTC"
Sep 17 14:05:05.914154 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.914146    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12350h36m7.366428637s"
Sep 17 14:05:05.959775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.959678    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:05.959946 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:05.959848    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:05.969055 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.969009    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2lrfm" event={"ID":"9c5ef9a9-f846-49ad-885b-dba324eea698","Type":"ContainerStarted","Data":"4bdb1436fb3f4e09e64243b30fd4ee9af961c64bc25454206e5bc319d5f671a3"}
Sep 17 14:05:05.970634 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.970601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5" event={"ID":"1672da8f-e4d5-4ca6-94d4-f9256ef75bda","Type":"ContainerStarted","Data":"6bea957bce633eb9767fbc4cb591e1bf8cbc5ccc648c7441878179a510c9bc70"}
Sep 17 14:05:05.972144 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.972111    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj" event={"ID":"9ae503a2-b685-4d50-96be-ae40f573337d","Type":"ContainerStarted","Data":"705134b9e565887ba5d5d4c719658d3fdf31c63ae179e9d6d44b2550d45bb2ef"}
Sep 17 14:05:05.974670 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.974638    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal" event={"ID":"f2a8032d1e1ba5143915f9592a048d25","Type":"ContainerStarted","Data":"4ec0bd46e6908319e45c5e4f90d26d9b6cd660155aaea767038e1dc316771887"}
Sep 17 14:05:05.976110 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.976083    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gc5s9" event={"ID":"26394392-22b2-42a8-98b8-a6b6f194f569","Type":"ContainerStarted","Data":"4ea6434ae669948aad2f56385434df41909d0bc8498e6ec8ccc9c5075fce73ef"}
Sep 17 14:05:05.977594 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.977561    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cns4" event={"ID":"c1c3e777-dd90-4fea-9735-157ef5f2ebb7","Type":"ContainerStarted","Data":"6fd18ce722371abe006ca2ff8e4ebb220e13ba40a521df3c602fd8bafed5efc8"}
Sep 17 14:05:05.978997 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.978970    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fnq7p" event={"ID":"644ea3fc-db18-4797-8d33-246b855769c9","Type":"ContainerStarted","Data":"d0c0015a8ac3a6d34b0040fc0400a09d5ee85a579cd5d0197014a7288d8e4088"}
Sep 17 14:05:05.980684 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.980660    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h2l92" event={"ID":"d18eac27-3f6f-4757-94c9-0e01cd87d96a","Type":"ContainerStarted","Data":"5ab8d3abdff7cc50b417cd6e995e902b65101a7ac14ed2d63bc41347ecb3b872"}
Sep 17 14:05:05.982051 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.982011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wr9bq" event={"ID":"29b2af19-f2bf-42d3-9716-639cb0848d7b","Type":"ContainerStarted","Data":"33fe9716d2b001948bdae9280c8b8580692689d721a22d7dca9f899d1a62dba5"}
Sep 17 14:05:05.986290 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.984350    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wjnk2" event={"ID":"9aabfb62-00a4-4dca-8796-ee6bcd6a1277","Type":"ContainerStarted","Data":"b17b3a59a2dbebefa0828f1bb0f01ce75f81cab7e5db079f3f05b3c712ca1713"}
Sep 17 14:05:05.998043 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:05.997996    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-146.ec2.internal" podStartSLOduration=1.9979769410000001 podStartE2EDuration="1.997976941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:05.996944579 +0000 UTC m=+3.588120356" watchObservedRunningTime="2026-09-17 14:05:05.997976941 +0000 UTC m=+3.589152710"
Sep 17 14:05:06.460871 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:06.460825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:06.461066 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:06.460897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:06.461137 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:06.461076    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:06.461201 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:06.461149    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret podName:3839cd65-b24b-4e79-b075-d703429c694f nodeName:}" failed. No retries permitted until 2026-09-17 14:05:08.46112616 +0000 UTC m=+6.052301905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret") pod "global-pull-secret-syncer-52kgw" (UID: "3839cd65-b24b-4e79-b075-d703429c694f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:06.461641 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:06.461620    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:06.461741 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:06.461681    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs podName:796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e nodeName:}" failed. No retries permitted until 2026-09-17 14:05:08.461663513 +0000 UTC m=+6.052839273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs") pod "network-metrics-daemon-m4bth" (UID: "796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:06.663567 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:06.663518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkmn\" (UniqueName: \"kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn\") pod \"network-check-target-lpgsd\" (UID: \"582bd8f2-b198-4a29-96d1-a838e1dad55c\") " pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:06.663755 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:06.663704    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:05:06.663755 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:06.663727    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:05:06.663755 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:06.663740    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdkmn for pod openshift-network-diagnostics/network-check-target-lpgsd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:06.663939 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:06.663808    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn podName:582bd8f2-b198-4a29-96d1-a838e1dad55c nodeName:}" failed. No retries permitted until 2026-09-17 14:05:08.663786071 +0000 UTC m=+6.254961821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdkmn" (UniqueName: "kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn") pod "network-check-target-lpgsd" (UID: "582bd8f2-b198-4a29-96d1-a838e1dad55c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:06.914923 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:06.914776    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 14:00:05 +0000 UTC" deadline="2028-03-29 13:51:13.243827502 +0000 UTC"
Sep 17 14:05:06.914923 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:06.914846    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13415h46m6.328986414s"
Sep 17 14:05:06.960127 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:06.960099    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:06.960296 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:06.960262    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:06.962616 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:06.960809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:06.962616 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:06.960935    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:07.959692 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:07.959654    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:07.960225 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:07.959808    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:08.480333 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:08.480290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:08.480545 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:08.480355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:08.480612 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:08.480543    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:08.480680 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:08.480616    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret podName:3839cd65-b24b-4e79-b075-d703429c694f nodeName:}" failed. No retries permitted until 2026-09-17 14:05:12.480594887 +0000 UTC m=+10.071770646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret") pod "global-pull-secret-syncer-52kgw" (UID: "3839cd65-b24b-4e79-b075-d703429c694f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:08.481155 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:08.481118    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:08.481259 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:08.481174    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs podName:796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e nodeName:}" failed. No retries permitted until 2026-09-17 14:05:12.481158902 +0000 UTC m=+10.072334643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs") pod "network-metrics-daemon-m4bth" (UID: "796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:08.682987 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:08.682943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkmn\" (UniqueName: \"kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn\") pod \"network-check-target-lpgsd\" (UID: \"582bd8f2-b198-4a29-96d1-a838e1dad55c\") " pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:08.683185 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:08.683131    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:05:08.683185 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:08.683155    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:05:08.683185 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:08.683169    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdkmn for pod openshift-network-diagnostics/network-check-target-lpgsd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:08.683363 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:08.683236    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn podName:582bd8f2-b198-4a29-96d1-a838e1dad55c nodeName:}" failed. No retries permitted until 2026-09-17 14:05:12.683214629 +0000 UTC m=+10.274390374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdkmn" (UniqueName: "kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn") pod "network-check-target-lpgsd" (UID: "582bd8f2-b198-4a29-96d1-a838e1dad55c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:08.962752 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:08.962656    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:08.963233 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:08.962812    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:08.963394 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:08.963236    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:08.963394 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:08.963333    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:09.959865 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:09.959827    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:09.960042 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:09.960001    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:10.960114 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:10.960079    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:10.960651 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:10.960101    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:10.960651 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:10.960232    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:10.960931 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:10.960433    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:11.959304 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:11.959258    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:11.959509 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:11.959435    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:12.521267 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:12.521218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:12.521898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:12.521291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:12.521898 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:12.521478    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:12.521898 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:12.521544    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret podName:3839cd65-b24b-4e79-b075-d703429c694f nodeName:}" failed. No retries permitted until 2026-09-17 14:05:20.521525661 +0000 UTC m=+18.112701416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret") pod "global-pull-secret-syncer-52kgw" (UID: "3839cd65-b24b-4e79-b075-d703429c694f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:12.522064 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:12.521941    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:12.522064 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:12.521988    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs podName:796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e nodeName:}" failed. No retries permitted until 2026-09-17 14:05:20.521973168 +0000 UTC m=+18.113148918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs") pod "network-metrics-daemon-m4bth" (UID: "796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:12.723223 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:12.723183    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkmn\" (UniqueName: \"kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn\") pod \"network-check-target-lpgsd\" (UID: \"582bd8f2-b198-4a29-96d1-a838e1dad55c\") " pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:12.723412 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:12.723300    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:05:12.723412 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:12.723324    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:05:12.723412 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:12.723335    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdkmn for pod openshift-network-diagnostics/network-check-target-lpgsd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:12.723412 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:12.723412    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn podName:582bd8f2-b198-4a29-96d1-a838e1dad55c nodeName:}" failed. No retries permitted until 2026-09-17 14:05:20.723394108 +0000 UTC m=+18.314569860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdkmn" (UniqueName: "kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn") pod "network-check-target-lpgsd" (UID: "582bd8f2-b198-4a29-96d1-a838e1dad55c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:12.960647 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:12.960551    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:12.960817 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:12.960667    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:12.960817 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:12.960716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:12.960918 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:12.960818    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:13.959841 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:13.959750    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:13.960322 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:13.959888    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:14.959563 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:14.959525    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:14.959735 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:14.959566    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:14.959735 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:14.959666    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:14.960067 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:14.960040    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:15.959712 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:15.959660    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:15.959912 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:15.959804    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:16.959159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:16.959115    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:16.959735 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:16.959115    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:16.959735 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:16.959255    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:16.959735 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:16.959329    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:17.959589 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:17.959540    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:17.960019 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:17.959699    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:18.959093 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:18.959051    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:18.959267 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:18.959191    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:18.959267 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:18.959239    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:18.959414 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:18.959383    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:19.959983 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:19.959937    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:19.960485 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:19.960083    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:20.580081 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:20.579995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:20.580257 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:20.580168    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:20.580257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:20.580179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:20.580257 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:20.580234    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret podName:3839cd65-b24b-4e79-b075-d703429c694f nodeName:}" failed. No retries permitted until 2026-09-17 14:05:36.5802152 +0000 UTC m=+34.171390961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret") pod "global-pull-secret-syncer-52kgw" (UID: "3839cd65-b24b-4e79-b075-d703429c694f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:20.580400 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:20.580326    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:20.580462 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:20.580408    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs podName:796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e nodeName:}" failed. No retries permitted until 2026-09-17 14:05:36.580390226 +0000 UTC m=+34.171565986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs") pod "network-metrics-daemon-m4bth" (UID: "796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:20.781073 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:20.781026    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkmn\" (UniqueName: \"kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn\") pod \"network-check-target-lpgsd\" (UID: \"582bd8f2-b198-4a29-96d1-a838e1dad55c\") " pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:20.781242 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:20.781198    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:05:20.781242 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:20.781229    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:05:20.781242 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:20.781244    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdkmn for pod openshift-network-diagnostics/network-check-target-lpgsd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:20.781403 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:20.781311    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn podName:582bd8f2-b198-4a29-96d1-a838e1dad55c nodeName:}" failed. No retries permitted until 2026-09-17 14:05:36.781291424 +0000 UTC m=+34.372467169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdkmn" (UniqueName: "kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn") pod "network-check-target-lpgsd" (UID: "582bd8f2-b198-4a29-96d1-a838e1dad55c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:20.959867 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:20.959777    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:20.960024 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:20.959777    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:20.960024 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:20.959926    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:20.960024 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:20.959996    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:21.959094 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:21.959058    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:21.959290 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:21.959193    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:22.960245 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:22.960203    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:22.960245 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:22.960249    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:22.960766 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:22.960332    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:22.960766 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:22.960450    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:23.959398 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:23.959344    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:23.959571 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:23.959520    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:24.960067 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:24.959681    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:24.960924 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:24.959727    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:24.960924 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:24.960193    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:24.960924 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:24.960263    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:25.039579 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.039540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gc5s9" event={"ID":"26394392-22b2-42a8-98b8-a6b6f194f569","Type":"ContainerStarted","Data":"b5aa4fe77fc0b724f29dae1dce657fc7b81014f5517bfa4eea59c3f8958089ff"}
Sep 17 14:05:25.041432 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.041395    2560 generic.go:356] "Generic (PLEG): container finished" podID="c1c3e777-dd90-4fea-9735-157ef5f2ebb7" containerID="1fc9e43ee3ae70faafdb01cfd4081dfcbecc94af62ccb8daa07496f35153b43d" exitCode=0
Sep 17 14:05:25.041563 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.041455    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cns4" event={"ID":"c1c3e777-dd90-4fea-9735-157ef5f2ebb7","Type":"ContainerDied","Data":"1fc9e43ee3ae70faafdb01cfd4081dfcbecc94af62ccb8daa07496f35153b43d"}
Sep 17 14:05:25.043562 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.043304    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fnq7p" event={"ID":"644ea3fc-db18-4797-8d33-246b855769c9","Type":"ContainerStarted","Data":"5a309903e489fab19f9715d99929bec784df901bf0f3a5ac13f8920fcc36bea8"}
Sep 17 14:05:25.045277 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.045255    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h2l92" event={"ID":"d18eac27-3f6f-4757-94c9-0e01cd87d96a","Type":"ContainerStarted","Data":"d573919c47e73a895fa5569e41c2de2c936a463c50babc543fda123853106d65"}
Sep 17 14:05:25.047077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.047040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wr9bq" event={"ID":"29b2af19-f2bf-42d3-9716-639cb0848d7b","Type":"ContainerStarted","Data":"07ab73a68d226126f684a34fff1188a12ca69cd19ab2d2638ca6b36185d6c99a"}
Sep 17 14:05:25.048745 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.048688    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wjnk2" event={"ID":"9aabfb62-00a4-4dca-8796-ee6bcd6a1277","Type":"ContainerStarted","Data":"49fed240e0f93a9a13d523c02c2694398db83f26286b1ef88c7bb4e7456a9bac"}
Sep 17 14:05:25.050407 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.050379    2560 generic.go:356] "Generic (PLEG): container finished" podID="e598cbcd04df48c89405a63ef0a1f2f5" containerID="54e33d133146114641312bab62b9d7be1f6f13c04639c158e196cbefa1e5aa6e" exitCode=0
Sep 17 14:05:25.050510 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.050410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal" event={"ID":"e598cbcd04df48c89405a63ef0a1f2f5","Type":"ContainerDied","Data":"54e33d133146114641312bab62b9d7be1f6f13c04639c158e196cbefa1e5aa6e"}
Sep 17 14:05:25.052264 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.052237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5" event={"ID":"1672da8f-e4d5-4ca6-94d4-f9256ef75bda","Type":"ContainerStarted","Data":"2b89274e327e226a7c45b05c4434f773ba1d5ed9496229f4fc0d4b50399b0e2d"}
Sep 17 14:05:25.055888 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.055841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj" event={"ID":"9ae503a2-b685-4d50-96be-ae40f573337d","Type":"ContainerStarted","Data":"153128463af0a86a03c3a36f99dff2a842b7a2d024178a454d2555faa5815d3d"}
Sep 17 14:05:25.055988 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.055934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj" event={"ID":"9ae503a2-b685-4d50-96be-ae40f573337d","Type":"ContainerStarted","Data":"a75d3f96977de7e7dc66e703629b740e9a207e4aedd563633a7f5367e1d22f09"}
Sep 17 14:05:25.056048 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.055987    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj" event={"ID":"9ae503a2-b685-4d50-96be-ae40f573337d","Type":"ContainerStarted","Data":"cc16c8deab9873f758a8f33cc41874655e4ab08a51e4534bb58a83d0f4518fc4"}
Sep 17 14:05:25.056107 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.056035    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj" event={"ID":"9ae503a2-b685-4d50-96be-ae40f573337d","Type":"ContainerStarted","Data":"af876df1d02b023ae1cbeecf44525a67acf107e485ae7751a2bba80dd9a78b8e"}
Sep 17 14:05:25.056107 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.056094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj" event={"ID":"9ae503a2-b685-4d50-96be-ae40f573337d","Type":"ContainerStarted","Data":"0f0c1ceffbead4b7678127d926657da0e9f6252b3c229b8230abd3521b33b7e8"}
Sep 17 14:05:25.056204 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.056142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj" event={"ID":"9ae503a2-b685-4d50-96be-ae40f573337d","Type":"ContainerStarted","Data":"df5d80828efb8ebdb1dc176d9c860b2c481b3b2d61e71c431af9921d201c336e"}
Sep 17 14:05:25.056607 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.056505    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gc5s9" podStartSLOduration=3.641404778 podStartE2EDuration="22.056487787s" totalImagesPullingTime="18.415083009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:25.055954997 +0000 UTC m=+22.647130761" watchObservedRunningTime="2026-09-17 14:05:25.056487787 +0000 UTC m=+22.647663723"
Sep 17 14:05:25.076076 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.076001    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h2l92" podStartSLOduration=3.638380088 podStartE2EDuration="22.075980253s" totalImagesPullingTime="18.437600165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:25.075863563 +0000 UTC m=+22.667039326" watchObservedRunningTime="2026-09-17 14:05:25.075980253 +0000 UTC m=+22.667156015"
Sep 17 14:05:25.092729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.092669    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wjnk2" podStartSLOduration=3.605356965 podStartE2EDuration="22.092648925s" totalImagesPullingTime="18.48729196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:25.092234628 +0000 UTC m=+22.683410392" watchObservedRunningTime="2026-09-17 14:05:25.092648925 +0000 UTC m=+22.683824687"
Sep 17 14:05:25.112567 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.112164    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wr9bq" podStartSLOduration=7.5214656 podStartE2EDuration="22.112143039s" totalImagesPullingTime="14.590677439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:25.112128299 +0000 UTC m=+22.703304062" watchObservedRunningTime="2026-09-17 14:05:25.112143039 +0000 UTC m=+22.703318802"
Sep 17 14:05:25.131786 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.131736    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fnq7p" podStartSLOduration=3.696656592 podStartE2EDuration="22.131716571s" totalImagesPullingTime="18.435059979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:25.130858882 +0000 UTC m=+22.722034646" watchObservedRunningTime="2026-09-17 14:05:25.131716571 +0000 UTC m=+22.722892335"
Sep 17 14:05:25.853118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.853081    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 14:05:25.890926 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.890771    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T14:05:25.853110364Z","UUID":"18472538-7f8e-4248-bb57-8867629f3423","Handler":null,"Name":"","Endpoint":""}
Sep 17 14:05:25.894399 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.892992    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 14:05:25.894399 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.893232    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 14:05:25.959858 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:25.959768    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:25.960008 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:25.959883    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:26.060235 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:26.060197    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal" event={"ID":"e598cbcd04df48c89405a63ef0a1f2f5","Type":"ContainerStarted","Data":"98a6fbb7a39e0208894944ae5299fe521c09c7c132a7a9bab9186a7c00514b77"}
Sep 17 14:05:26.062129 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:26.062098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2lrfm" event={"ID":"9c5ef9a9-f846-49ad-885b-dba324eea698","Type":"ContainerStarted","Data":"42b40e4a9ce1d8866a2ad5d0861be736e5e482a5dedf62b7bbde0079690e216b"}
Sep 17 14:05:26.064203 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:26.064175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5" event={"ID":"1672da8f-e4d5-4ca6-94d4-f9256ef75bda","Type":"ContainerStarted","Data":"810013cb4215f821903b379b8eb288bb4054a6e92f9a682951330537be459119"}
Sep 17 14:05:26.115750 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:26.115686    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-146.ec2.internal" podStartSLOduration=22.115671028 podStartE2EDuration="22.115671028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:26.115520699 +0000 UTC m=+23.706696461" watchObservedRunningTime="2026-09-17 14:05:26.115671028 +0000 UTC m=+23.706846791"
Sep 17 14:05:26.135978 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:26.135932    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2lrfm" podStartSLOduration=4.6817852129999995 podStartE2EDuration="23.135916615s" totalImagesPullingTime="18.454131402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:26.135583472 +0000 UTC m=+23.726759237" watchObservedRunningTime="2026-09-17 14:05:26.135916615 +0000 UTC m=+23.727092376"
Sep 17 14:05:26.683122 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:26.683061    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:26.683863 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:26.683839    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:26.959435 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:26.959402    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:26.959543 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:26.959433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:26.959587 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:26.959542    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:26.959691 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:26.959665    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:27.070835 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:27.070789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj" event={"ID":"9ae503a2-b685-4d50-96be-ae40f573337d","Type":"ContainerStarted","Data":"906e50b2b77f9d848ca19b6237212dfaf411ccc4709e6eeb03feea5359049c05"}
Sep 17 14:05:27.071338 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:27.071049    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:27.071608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:27.071587    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wr9bq"
Sep 17 14:05:27.959432 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:27.959396    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:27.959616 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:27.959528    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:28.075971 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:28.075933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5" event={"ID":"1672da8f-e4d5-4ca6-94d4-f9256ef75bda","Type":"ContainerStarted","Data":"36ea9e22f026befcf4cba6b5c729e8b64750644c50a27ff5392dfa2ae56442ae"}
Sep 17 14:05:28.099150 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:28.099089    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qf5" podStartSLOduration=4.343562308 podStartE2EDuration="25.099071377s" totalImagesPullingTime="20.755509069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:28.097782699 +0000 UTC m=+25.688958461" watchObservedRunningTime="2026-09-17 14:05:28.099071377 +0000 UTC m=+25.690247140"
Sep 17 14:05:28.959670 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:28.959627    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:28.959863 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:28.959764    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:28.959863 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:28.959787    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:28.959982 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:28.959896    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:29.959818 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:29.959580    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:29.960186 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:29.959862    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:30.084795 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:30.084764    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj" event={"ID":"9ae503a2-b685-4d50-96be-ae40f573337d","Type":"ContainerStarted","Data":"5f4f5840e71766df26269a853dc149a562cd3255ebc2fd613acbaca1287ef192"}
Sep 17 14:05:30.085097 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:30.084982    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:30.086809 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:30.086772    2560 generic.go:356] "Generic (PLEG): container finished" podID="c1c3e777-dd90-4fea-9735-157ef5f2ebb7" containerID="a58f918ffc5d34e9a26485d2a1484a8949c18a109954e4647ae12b04546003f6" exitCode=0
Sep 17 14:05:30.086939 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:30.086832    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cns4" event={"ID":"c1c3e777-dd90-4fea-9735-157ef5f2ebb7","Type":"ContainerDied","Data":"a58f918ffc5d34e9a26485d2a1484a8949c18a109954e4647ae12b04546003f6"}
Sep 17 14:05:30.100673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:30.100644    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:30.114953 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:30.114904    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj" podStartSLOduration=8.604015724 podStartE2EDuration="27.114888017s" totalImagesPullingTime="18.510872293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:30.112983719 +0000 UTC m=+27.704159503" watchObservedRunningTime="2026-09-17 14:05:30.114888017 +0000 UTC m=+27.706063810"
Sep 17 14:05:30.959762 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:30.959583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:30.959762 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:30.959603    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:30.959762 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:30.959743    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:30.960480 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:30.959845    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:31.092558 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.092488    2560 generic.go:356] "Generic (PLEG): container finished" podID="c1c3e777-dd90-4fea-9735-157ef5f2ebb7" containerID="811d57afbc6431d3235221bac7af796738dc826eec84fe31a036550b4981b743" exitCode=0
Sep 17 14:05:31.092690 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.092565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cns4" event={"ID":"c1c3e777-dd90-4fea-9735-157ef5f2ebb7","Type":"ContainerDied","Data":"811d57afbc6431d3235221bac7af796738dc826eec84fe31a036550b4981b743"}
Sep 17 14:05:31.092747 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.092719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:31.092824 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.092811    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:31.110528 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.110498    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:05:31.367295 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.367208    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-52kgw"]
Sep 17 14:05:31.367446 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.367311    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:31.367446 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:31.367408    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:31.370190 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.370161    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lpgsd"]
Sep 17 14:05:31.370480 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.370458    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:31.370596 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:31.370570    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:31.370919 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.370902    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m4bth"]
Sep 17 14:05:31.371019 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:31.370992    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:31.371113 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:31.371097    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:32.960400 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:32.960147    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:32.961077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:32.960212    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:32.961077 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:32.960480    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:32.961077 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:32.960591    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:32.961077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:32.960242    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:32.961077 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:32.960726    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:33.099116 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:33.099088    2560 generic.go:356] "Generic (PLEG): container finished" podID="c1c3e777-dd90-4fea-9735-157ef5f2ebb7" containerID="9cff6e3b3ff48123bfc9c43accb6d45c0b07949d536e241ece85f4cd426c671c" exitCode=0
Sep 17 14:05:33.099253 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:33.099170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cns4" event={"ID":"c1c3e777-dd90-4fea-9735-157ef5f2ebb7","Type":"ContainerDied","Data":"9cff6e3b3ff48123bfc9c43accb6d45c0b07949d536e241ece85f4cd426c671c"}
Sep 17 14:05:34.959968 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:34.959932    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:34.960430 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:34.959932    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:34.960430 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:34.960066    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:34.960430 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:34.960171    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:34.960430 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:34.960213    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:34.960430 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:34.960320    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:36.594189 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:36.594149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:36.594763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:36.594196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:36.594763 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.594313    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:36.594763 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.594317    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:36.594763 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.594389    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret podName:3839cd65-b24b-4e79-b075-d703429c694f nodeName:}" failed. No retries permitted until 2026-09-17 14:06:08.594372125 +0000 UTC m=+66.185547884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret") pod "global-pull-secret-syncer-52kgw" (UID: "3839cd65-b24b-4e79-b075-d703429c694f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:05:36.594763 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.594403    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs podName:796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e nodeName:}" failed. No retries permitted until 2026-09-17 14:06:08.594396594 +0000 UTC m=+66.185572334 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs") pod "network-metrics-daemon-m4bth" (UID: "796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:05:36.796435 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:36.796396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkmn\" (UniqueName: \"kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn\") pod \"network-check-target-lpgsd\" (UID: \"582bd8f2-b198-4a29-96d1-a838e1dad55c\") " pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:36.796666 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.796583    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:05:36.796666 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.796610    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:05:36.796666 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.796625    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdkmn for pod openshift-network-diagnostics/network-check-target-lpgsd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:36.796815 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.796687    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn podName:582bd8f2-b198-4a29-96d1-a838e1dad55c nodeName:}" failed. No retries permitted until 2026-09-17 14:06:08.796672261 +0000 UTC m=+66.387848002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdkmn" (UniqueName: "kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn") pod "network-check-target-lpgsd" (UID: "582bd8f2-b198-4a29-96d1-a838e1dad55c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:05:36.959566 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:36.959450    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:36.959566 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:36.959450    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:36.959881 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.959585    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52kgw" podUID="3839cd65-b24b-4e79-b075-d703429c694f"
Sep 17 14:05:36.959881 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:36.959605    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:36.959881 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.959720    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpgsd" podUID="582bd8f2-b198-4a29-96d1-a838e1dad55c"
Sep 17 14:05:36.959881 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:36.959825    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4bth" podUID="796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e"
Sep 17 14:05:37.215648 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.215568    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-146.ec2.internal" event="NodeReady"
Sep 17 14:05:37.215796 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.215729    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 14:05:37.265033 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.264994    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65bb67c7d5-svssp"]
Sep 17 14:05:37.283408 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.283346    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hm4vd"]
Sep 17 14:05:37.283592 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.283510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.286205 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.286175    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 14:05:37.286205 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.286194    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 14:05:37.286416 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.286284    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 14:05:37.286544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.286528    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sjbd6\""
Sep 17 14:05:37.292325 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.292296    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 14:05:37.297482 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.297439    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-klkvh"]
Sep 17 14:05:37.297632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.297615    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.300286 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.300263    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 14:05:37.301008 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.300828    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 14:05:37.301008 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.300921    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hcwc9\""
Sep 17 14:05:37.314092 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.314064    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hm4vd"]
Sep 17 14:05:37.314232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.314113    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-klkvh"]
Sep 17 14:05:37.314232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.314135    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65bb67c7d5-svssp"]
Sep 17 14:05:37.314232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.314201    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-klkvh"
Sep 17 14:05:37.317415 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.317220    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 14:05:37.317415 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.317293    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-84cnx\""
Sep 17 14:05:37.317415 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.317299    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 14:05:37.317881 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.317864    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 14:05:37.400592 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpts6\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-kube-api-access-xpts6\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.400592 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c43836f-f444-4fa9-9155-b75f04b32d80-config-volume\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.400791 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.400791 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400664    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.400791 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-bound-sa-token\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.400791 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400720    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/823cdcd0-1f52-455f-98aa-1db092020587-cert\") pod \"ingress-canary-klkvh\" (UID: \"823cdcd0-1f52-455f-98aa-1db092020587\") " pod="openshift-ingress-canary/ingress-canary-klkvh"
Sep 17 14:05:37.400791 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400755    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-image-registry-private-configuration\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.400938 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c43836f-f444-4fa9-9155-b75f04b32d80-tmp-dir\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.400938 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-trusted-ca\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.400938 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-installation-pull-secrets\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.400938 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdqw\" (UniqueName: \"kubernetes.io/projected/6c43836f-f444-4fa9-9155-b75f04b32d80-kube-api-access-4jdqw\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.401053 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2q65\" (UniqueName: \"kubernetes.io/projected/823cdcd0-1f52-455f-98aa-1db092020587-kube-api-access-x2q65\") pod \"ingress-canary-klkvh\" (UID: \"823cdcd0-1f52-455f-98aa-1db092020587\") " pod="openshift-ingress-canary/ingress-canary-klkvh"
Sep 17 14:05:37.401053 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.400991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c381f96e-b724-4232-a4ca-f06476d465a1-ca-trust-extracted\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.401053 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.401007    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-registry-certificates\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.501933 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.501894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-registry-certificates\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.501933 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.501939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpts6\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-kube-api-access-xpts6\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.502172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.501955    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c43836f-f444-4fa9-9155-b75f04b32d80-config-volume\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.502172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.501970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.502172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.502172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502023    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-bound-sa-token\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.502172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502038    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/823cdcd0-1f52-455f-98aa-1db092020587-cert\") pod \"ingress-canary-klkvh\" (UID: \"823cdcd0-1f52-455f-98aa-1db092020587\") " pod="openshift-ingress-canary/ingress-canary-klkvh"
Sep 17 14:05:37.502172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502066    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-image-registry-private-configuration\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.502172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502090    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c43836f-f444-4fa9-9155-b75f04b32d80-tmp-dir\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.502172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-trusted-ca\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.502172 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:37.502153    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:05:37.502172 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:37.502172    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb67c7d5-svssp: secret "image-registry-tls" not found
Sep 17 14:05:37.502714 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:37.502176    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:05:37.502714 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:37.502232    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls podName:c381f96e-b724-4232-a4ca-f06476d465a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:38.002213529 +0000 UTC m=+35.593389277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls") pod "image-registry-65bb67c7d5-svssp" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1") : secret "image-registry-tls" not found
Sep 17 14:05:37.502714 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:37.502251    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls podName:6c43836f-f444-4fa9-9155-b75f04b32d80 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:38.002241215 +0000 UTC m=+35.593416959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls") pod "dns-default-hm4vd" (UID: "6c43836f-f444-4fa9-9155-b75f04b32d80") : secret "dns-default-metrics-tls" not found
Sep 17 14:05:37.502714 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-installation-pull-secrets\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.502714 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdqw\" (UniqueName: \"kubernetes.io/projected/6c43836f-f444-4fa9-9155-b75f04b32d80-kube-api-access-4jdqw\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.502714 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2q65\" (UniqueName: \"kubernetes.io/projected/823cdcd0-1f52-455f-98aa-1db092020587-kube-api-access-x2q65\") pod \"ingress-canary-klkvh\" (UID: \"823cdcd0-1f52-455f-98aa-1db092020587\") " pod="openshift-ingress-canary/ingress-canary-klkvh"
Sep 17 14:05:37.502714 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c381f96e-b724-4232-a4ca-f06476d465a1-ca-trust-extracted\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.502714 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502518    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c43836f-f444-4fa9-9155-b75f04b32d80-tmp-dir\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.502714 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c43836f-f444-4fa9-9155-b75f04b32d80-config-volume\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.502714 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-registry-certificates\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.503107 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.502933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c381f96e-b724-4232-a4ca-f06476d465a1-ca-trust-extracted\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.503491 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.503471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-trusted-ca\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.506499 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.506478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-image-registry-private-configuration\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.506616 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.506498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-installation-pull-secrets\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.506616 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.506537    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/823cdcd0-1f52-455f-98aa-1db092020587-cert\") pod \"ingress-canary-klkvh\" (UID: \"823cdcd0-1f52-455f-98aa-1db092020587\") " pod="openshift-ingress-canary/ingress-canary-klkvh"
Sep 17 14:05:37.516464 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.516440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdqw\" (UniqueName: \"kubernetes.io/projected/6c43836f-f444-4fa9-9155-b75f04b32d80-kube-api-access-4jdqw\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:37.516571 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.516443    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-bound-sa-token\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.516571 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.516517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2q65\" (UniqueName: \"kubernetes.io/projected/823cdcd0-1f52-455f-98aa-1db092020587-kube-api-access-x2q65\") pod \"ingress-canary-klkvh\" (UID: \"823cdcd0-1f52-455f-98aa-1db092020587\") " pod="openshift-ingress-canary/ingress-canary-klkvh"
Sep 17 14:05:37.516757 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.516740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpts6\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-kube-api-access-xpts6\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:37.624650 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:37.624619    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-klkvh"
Sep 17 14:05:38.005855 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.005635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:38.006028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.005875    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:38.006028 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:38.005783    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:05:38.006028 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:38.005967    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls podName:6c43836f-f444-4fa9-9155-b75f04b32d80 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:39.005950136 +0000 UTC m=+36.597125892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls") pod "dns-default-hm4vd" (UID: "6c43836f-f444-4fa9-9155-b75f04b32d80") : secret "dns-default-metrics-tls" not found
Sep 17 14:05:38.006197 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:38.006029    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:05:38.006197 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:38.006044    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb67c7d5-svssp: secret "image-registry-tls" not found
Sep 17 14:05:38.006197 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:38.006098    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls podName:c381f96e-b724-4232-a4ca-f06476d465a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:39.006082179 +0000 UTC m=+36.597257934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls") pod "image-registry-65bb67c7d5-svssp" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1") : secret "image-registry-tls" not found
Sep 17 14:05:38.078660 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.078580    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-klkvh"]
Sep 17 14:05:38.091322 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:05:38.091280    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823cdcd0_1f52_455f_98aa_1db092020587.slice/crio-bac616dba0ab15e4e9c4fe0321eb17f783b41a5c84a1177c2b0f5879da1d82af WatchSource:0}: Error finding container bac616dba0ab15e4e9c4fe0321eb17f783b41a5c84a1177c2b0f5879da1d82af: Status 404 returned error can't find the container with id bac616dba0ab15e4e9c4fe0321eb17f783b41a5c84a1177c2b0f5879da1d82af
Sep 17 14:05:38.112814 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.112784    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-klkvh" event={"ID":"823cdcd0-1f52-455f-98aa-1db092020587","Type":"ContainerStarted","Data":"bac616dba0ab15e4e9c4fe0321eb17f783b41a5c84a1177c2b0f5879da1d82af"}
Sep 17 14:05:38.116004 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.115973    2560 generic.go:356] "Generic (PLEG): container finished" podID="c1c3e777-dd90-4fea-9735-157ef5f2ebb7" containerID="9578e487c12e5945540ec3af93631280ce9bd60692c5c6feb065cda7c15b5968" exitCode=0
Sep 17 14:05:38.116109 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.116022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cns4" event={"ID":"c1c3e777-dd90-4fea-9735-157ef5f2ebb7","Type":"ContainerDied","Data":"9578e487c12e5945540ec3af93631280ce9bd60692c5c6feb065cda7c15b5968"}
Sep 17 14:05:38.959552 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.959509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:05:38.959552 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.959531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:05:38.960465 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.959510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:05:38.963569 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.963546    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 14:05:38.963713 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.963633    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-27m2f\""
Sep 17 14:05:38.963713 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.963662    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 14:05:38.963713 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.963688    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lkjmx\""
Sep 17 14:05:38.963888 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.963787    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 14:05:38.963986 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:38.963967    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 14:05:39.013202 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:39.013161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:39.013379 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:39.013217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:39.013379 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:39.013331    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:05:39.013379 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:39.013341    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:05:39.013501 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:39.013345    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb67c7d5-svssp: secret "image-registry-tls" not found
Sep 17 14:05:39.013501 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:39.013428    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls podName:6c43836f-f444-4fa9-9155-b75f04b32d80 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:41.013413529 +0000 UTC m=+38.604589269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls") pod "dns-default-hm4vd" (UID: "6c43836f-f444-4fa9-9155-b75f04b32d80") : secret "dns-default-metrics-tls" not found
Sep 17 14:05:39.013501 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:39.013468    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls podName:c381f96e-b724-4232-a4ca-f06476d465a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:41.013450478 +0000 UTC m=+38.604626236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls") pod "image-registry-65bb67c7d5-svssp" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1") : secret "image-registry-tls" not found
Sep 17 14:05:39.122448 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:39.122410    2560 generic.go:356] "Generic (PLEG): container finished" podID="c1c3e777-dd90-4fea-9735-157ef5f2ebb7" containerID="9233725815abeb4f3255014ae97b89079b4240ef431b252d4aa9a34d420c2a07" exitCode=0
Sep 17 14:05:39.122642 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:39.122463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cns4" event={"ID":"c1c3e777-dd90-4fea-9735-157ef5f2ebb7","Type":"ContainerDied","Data":"9233725815abeb4f3255014ae97b89079b4240ef431b252d4aa9a34d420c2a07"}
Sep 17 14:05:40.129519 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:40.129475    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7cns4" event={"ID":"c1c3e777-dd90-4fea-9735-157ef5f2ebb7","Type":"ContainerStarted","Data":"4bbe4df08eb275ffcd70f3547f08dc5e1b0031d486aa6719274ac75ece689b97"}
Sep 17 14:05:40.155958 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:40.155887    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7cns4" podStartSLOduration=7.748236919 podStartE2EDuration="37.155866489s" totalImagesPullingTime="29.40762957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:40.153955116 +0000 UTC m=+37.745130900" watchObservedRunningTime="2026-09-17 14:05:40.155866489 +0000 UTC m=+37.747042250"
Sep 17 14:05:41.029497 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:41.029397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:41.029497 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:41.029445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:41.029700 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:41.029552    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:05:41.029700 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:41.029608    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:05:41.029700 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:41.029619    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb67c7d5-svssp: secret "image-registry-tls" not found
Sep 17 14:05:41.029700 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:41.029628    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls podName:6c43836f-f444-4fa9-9155-b75f04b32d80 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:45.029612867 +0000 UTC m=+42.620788607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls") pod "dns-default-hm4vd" (UID: "6c43836f-f444-4fa9-9155-b75f04b32d80") : secret "dns-default-metrics-tls" not found
Sep 17 14:05:41.029700 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:41.029659    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls podName:c381f96e-b724-4232-a4ca-f06476d465a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:45.029646825 +0000 UTC m=+42.620822569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls") pod "image-registry-65bb67c7d5-svssp" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1") : secret "image-registry-tls" not found
Sep 17 14:05:41.133857 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:41.133816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-klkvh" event={"ID":"823cdcd0-1f52-455f-98aa-1db092020587","Type":"ContainerStarted","Data":"4a66383d6e483ddeaff9f725880cb8f2082fb1ca4a853a330f2e57a944981dda"}
Sep 17 14:05:41.162696 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:41.162632    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-klkvh" podStartSLOduration=1.521044731 podStartE2EDuration="4.162613913s" totalImagesPullingTime="2.641569182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:41.161974619 +0000 UTC m=+38.753150378" watchObservedRunningTime="2026-09-17 14:05:41.162613913 +0000 UTC m=+38.753789679"
Sep 17 14:05:44.388929 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.388711    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g6ww4"]
Sep 17 14:05:44.423018 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.422985    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g6ww4"]
Sep 17 14:05:44.423177 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.423112    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.426282 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.426259    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 14:05:44.426416 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.426298    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-48lqt\""
Sep 17 14:05:44.426796 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.426782    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 14:05:44.427268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.427252    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 14:05:44.429396 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.429374    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 14:05:44.558347 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.558307    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.558534 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.558400    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zzxp\" (UniqueName: \"kubernetes.io/projected/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-api-access-6zzxp\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.558534 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.558432    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40712e5d-89b9-4fcd-8561-769d4a59cd59-crio-socket\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.558534 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.558448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40712e5d-89b9-4fcd-8561-769d4a59cd59-data-volume\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.558534 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.558520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.659708 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.659633    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zzxp\" (UniqueName: \"kubernetes.io/projected/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-api-access-6zzxp\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.659708 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.659668    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40712e5d-89b9-4fcd-8561-769d4a59cd59-crio-socket\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.659708 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.659686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40712e5d-89b9-4fcd-8561-769d4a59cd59-data-volume\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.659890 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.659717    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.659890 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:44.659790    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:05:44.659890 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.659849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.660033 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:44.659907    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm podName:40712e5d-89b9-4fcd-8561-769d4a59cd59 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:45.159888575 +0000 UTC m=+42.751064318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g6ww4" (UID: "40712e5d-89b9-4fcd-8561-769d4a59cd59") : configmap "kube-rbac-proxy" not found
Sep 17 14:05:44.660033 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.659918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40712e5d-89b9-4fcd-8561-769d4a59cd59-crio-socket\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.660033 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:44.660015    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:05:44.660147 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.660058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40712e5d-89b9-4fcd-8561-769d4a59cd59-data-volume\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:44.660147 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:44.660082    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls podName:40712e5d-89b9-4fcd-8561-769d4a59cd59 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:45.160061271 +0000 UTC m=+42.751237029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g6ww4" (UID: "40712e5d-89b9-4fcd-8561-769d4a59cd59") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:05:44.671952 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:44.671928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zzxp\" (UniqueName: \"kubernetes.io/projected/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-api-access-6zzxp\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:45.062456 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:45.062425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:45.062627 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:45.062477    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:45.062627 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:45.062581    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:05:45.062627 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:45.062592    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:05:45.062627 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:45.062609    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb67c7d5-svssp: secret "image-registry-tls" not found
Sep 17 14:05:45.062751 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:45.062651    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls podName:6c43836f-f444-4fa9-9155-b75f04b32d80 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:53.062633765 +0000 UTC m=+50.653809531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls") pod "dns-default-hm4vd" (UID: "6c43836f-f444-4fa9-9155-b75f04b32d80") : secret "dns-default-metrics-tls" not found
Sep 17 14:05:45.062751 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:45.062665    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls podName:c381f96e-b724-4232-a4ca-f06476d465a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:53.062658961 +0000 UTC m=+50.653834700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls") pod "image-registry-65bb67c7d5-svssp" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1") : secret "image-registry-tls" not found
Sep 17 14:05:45.162863 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:45.162825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:45.163051 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:45.162910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:45.163051 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:45.162986    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:05:45.163137 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:45.163054    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls podName:40712e5d-89b9-4fcd-8561-769d4a59cd59 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:46.163038558 +0000 UTC m=+43.754214298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g6ww4" (UID: "40712e5d-89b9-4fcd-8561-769d4a59cd59") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:05:45.163137 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:45.162989    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:05:45.163137 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:45.163116    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm podName:40712e5d-89b9-4fcd-8561-769d4a59cd59 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:46.163104441 +0000 UTC m=+43.754280214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g6ww4" (UID: "40712e5d-89b9-4fcd-8561-769d4a59cd59") : configmap "kube-rbac-proxy" not found
Sep 17 14:05:46.170888 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:46.170849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:46.171285 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:46.170935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:46.171285 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:46.171005    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:05:46.171285 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:46.171006    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:05:46.171285 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:46.171068    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm podName:40712e5d-89b9-4fcd-8561-769d4a59cd59 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:48.171054636 +0000 UTC m=+45.762230376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g6ww4" (UID: "40712e5d-89b9-4fcd-8561-769d4a59cd59") : configmap "kube-rbac-proxy" not found
Sep 17 14:05:46.171285 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:46.171083    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls podName:40712e5d-89b9-4fcd-8561-769d4a59cd59 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:48.171075967 +0000 UTC m=+45.762251707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g6ww4" (UID: "40712e5d-89b9-4fcd-8561-769d4a59cd59") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:05:46.293418 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:46.293390    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gc5s9_26394392-22b2-42a8-98b8-a6b6f194f569/dns-node-resolver/0.log"
Sep 17 14:05:47.292260 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:47.292235    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h2l92_d18eac27-3f6f-4757-94c9-0e01cd87d96a/node-ca/0.log"
Sep 17 14:05:47.895344 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:47.895297    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-klkvh_823cdcd0-1f52-455f-98aa-1db092020587/serve-healthcheck-canary/0.log"
Sep 17 14:05:48.188098 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:48.187999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:48.188255 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:48.188098    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:48.188255 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:48.188154    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:05:48.188255 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:48.188182    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:05:48.188255 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:48.188228    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls podName:40712e5d-89b9-4fcd-8561-769d4a59cd59 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:52.188211602 +0000 UTC m=+49.779387347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g6ww4" (UID: "40712e5d-89b9-4fcd-8561-769d4a59cd59") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:05:48.188255 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:48.188244    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm podName:40712e5d-89b9-4fcd-8561-769d4a59cd59 nodeName:}" failed. No retries permitted until 2026-09-17 14:05:52.188237686 +0000 UTC m=+49.779413425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g6ww4" (UID: "40712e5d-89b9-4fcd-8561-769d4a59cd59") : configmap "kube-rbac-proxy" not found
Sep 17 14:05:52.220585 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:52.220542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:52.221015 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:52.220606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:52.221015 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:52.220714    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:05:52.221015 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:52.220767    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls podName:40712e5d-89b9-4fcd-8561-769d4a59cd59 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:00.220754013 +0000 UTC m=+57.811929767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g6ww4" (UID: "40712e5d-89b9-4fcd-8561-769d4a59cd59") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:05:52.221247 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:52.221095    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40712e5d-89b9-4fcd-8561-769d4a59cd59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:05:53.128402 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:53.128338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:05:53.128683 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:05:53.128426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:05:53.128683 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:53.128505    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:05:53.128683 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:53.128573    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 14:05:53.128683 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:53.128585    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls podName:6c43836f-f444-4fa9-9155-b75f04b32d80 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:09.128564145 +0000 UTC m=+66.719739910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls") pod "dns-default-hm4vd" (UID: "6c43836f-f444-4fa9-9155-b75f04b32d80") : secret "dns-default-metrics-tls" not found
Sep 17 14:05:53.128683 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:53.128591    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb67c7d5-svssp: secret "image-registry-tls" not found
Sep 17 14:05:53.128683 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:05:53.128642    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls podName:c381f96e-b724-4232-a4ca-f06476d465a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:09.128631157 +0000 UTC m=+66.719806917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls") pod "image-registry-65bb67c7d5-svssp" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1") : secret "image-registry-tls" not found
Sep 17 14:06:00.287194 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:00.287149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:06:00.289446 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:00.289423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40712e5d-89b9-4fcd-8561-769d4a59cd59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6ww4\" (UID: \"40712e5d-89b9-4fcd-8561-769d4a59cd59\") " pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:06:00.332593 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:00.332555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g6ww4"
Sep 17 14:06:00.618913 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:00.618833    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g6ww4"]
Sep 17 14:06:01.179926 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:01.179888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g6ww4" event={"ID":"40712e5d-89b9-4fcd-8561-769d4a59cd59","Type":"ContainerStarted","Data":"a1f83dec13c0a77d5404313db605c2b5ff54809de3212dc34dc6cb3aedd9a740"}
Sep 17 14:06:01.180090 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:01.179952    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g6ww4" event={"ID":"40712e5d-89b9-4fcd-8561-769d4a59cd59","Type":"ContainerStarted","Data":"529c71d854f06448e0b59bd2a908acbbac2c4b09788a781372274bc40608fd9e"}
Sep 17 14:06:02.184224 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:02.184187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g6ww4" event={"ID":"40712e5d-89b9-4fcd-8561-769d4a59cd59","Type":"ContainerStarted","Data":"24ea193b4504de5636c944454c00de8446ab7525d71792379cc3e93c9cfa10b6"}
Sep 17 14:06:03.117228 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:03.117197    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpjkj"
Sep 17 14:06:04.191565 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:04.191527    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g6ww4" event={"ID":"40712e5d-89b9-4fcd-8561-769d4a59cd59","Type":"ContainerStarted","Data":"d0214b0dbadc8795cf9b4937e417492ee5ea4e66a87d356aaf9b76ba869061c7"}
Sep 17 14:06:04.212622 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:04.212577    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g6ww4" podStartSLOduration=17.716330781 podStartE2EDuration="20.212560203s" totalImagesPullingTime="2.496229422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:04.211471711 +0000 UTC m=+61.802647485" watchObservedRunningTime="2026-09-17 14:06:04.212560203 +0000 UTC m=+61.803735964"
Sep 17 14:06:08.649276 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.649215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:06:08.649276 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.649285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:06:08.652300 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.652279    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 14:06:08.653057 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.653041    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 14:06:08.662299 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.662265    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3839cd65-b24b-4e79-b075-d703429c694f-original-pull-secret\") pod \"global-pull-secret-syncer-52kgw\" (UID: \"3839cd65-b24b-4e79-b075-d703429c694f\") " pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:06:08.662439 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.662416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e-metrics-certs\") pod \"network-metrics-daemon-m4bth\" (UID: \"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e\") " pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:06:08.671344 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.671324    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52kgw"
Sep 17 14:06:08.688549 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.688527    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-27m2f\""
Sep 17 14:06:08.696000 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.695978    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4bth"
Sep 17 14:06:08.850854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.850810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkmn\" (UniqueName: \"kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn\") pod \"network-check-target-lpgsd\" (UID: \"582bd8f2-b198-4a29-96d1-a838e1dad55c\") " pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:06:08.854761 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.854735    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 14:06:08.864606 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.864585    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 14:06:08.875349 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.875326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdkmn\" (UniqueName: \"kubernetes.io/projected/582bd8f2-b198-4a29-96d1-a838e1dad55c-kube-api-access-gdkmn\") pod \"network-check-target-lpgsd\" (UID: \"582bd8f2-b198-4a29-96d1-a838e1dad55c\") " pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:06:08.949675 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.949598    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-52kgw"]
Sep 17 14:06:08.952956 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:08.952927    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3839cd65_b24b_4e79_b075_d703429c694f.slice/crio-40b54c2417f39e570fc6f692b98dfc90db277653e089233389353fa0b3b220a6 WatchSource:0}: Error finding container 40b54c2417f39e570fc6f692b98dfc90db277653e089233389353fa0b3b220a6: Status 404 returned error can't find the container with id 40b54c2417f39e570fc6f692b98dfc90db277653e089233389353fa0b3b220a6
Sep 17 14:06:08.971134 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.971111    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m4bth"]
Sep 17 14:06:08.973681 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:08.973660    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796b6e4c_79f6_4a2b_98a5_eb2f8daeb09e.slice/crio-0cff78b9ee7b71e63cba0d31c045b5d093a70424f79607456582378f8a1305e2 WatchSource:0}: Error finding container 0cff78b9ee7b71e63cba0d31c045b5d093a70424f79607456582378f8a1305e2: Status 404 returned error can't find the container with id 0cff78b9ee7b71e63cba0d31c045b5d093a70424f79607456582378f8a1305e2
Sep 17 14:06:08.981617 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.981597    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lkjmx\""
Sep 17 14:06:08.989730 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:08.989713    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:06:09.153522 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.153483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:06:09.153522 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.153529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:06:09.155852 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.155831    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c43836f-f444-4fa9-9155-b75f04b32d80-metrics-tls\") pod \"dns-default-hm4vd\" (UID: \"6c43836f-f444-4fa9-9155-b75f04b32d80\") " pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:06:09.155934 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.155912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") pod \"image-registry-65bb67c7d5-svssp\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") " pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:06:09.204913 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.204831    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4bth" event={"ID":"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e","Type":"ContainerStarted","Data":"0cff78b9ee7b71e63cba0d31c045b5d093a70424f79607456582378f8a1305e2"}
Sep 17 14:06:09.205882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.205859    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-52kgw" event={"ID":"3839cd65-b24b-4e79-b075-d703429c694f","Type":"ContainerStarted","Data":"40b54c2417f39e570fc6f692b98dfc90db277653e089233389353fa0b3b220a6"}
Sep 17 14:06:09.267177 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.267145    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lpgsd"]
Sep 17 14:06:09.270597 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:09.270571    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582bd8f2_b198_4a29_96d1_a838e1dad55c.slice/crio-7c5f18c393b1de4becf0853014b2014d46b30c8200497b4b3d9444482b249cc3 WatchSource:0}: Error finding container 7c5f18c393b1de4becf0853014b2014d46b30c8200497b4b3d9444482b249cc3: Status 404 returned error can't find the container with id 7c5f18c393b1de4becf0853014b2014d46b30c8200497b4b3d9444482b249cc3
Sep 17 14:06:09.404377 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.404322    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sjbd6\""
Sep 17 14:06:09.410989 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.410959    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:06:09.413941 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.413921    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hcwc9\""
Sep 17 14:06:09.420510 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.420481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:06:09.727204 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.727110    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65bb67c7d5-svssp"]
Sep 17 14:06:09.732045 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:09.732010    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc381f96e_b724_4232_a4ca_f06476d465a1.slice/crio-5911af3ecd71a24bddc3bdbf4ca99965ec4bae25ec452edc8f99e96ac6acd360 WatchSource:0}: Error finding container 5911af3ecd71a24bddc3bdbf4ca99965ec4bae25ec452edc8f99e96ac6acd360: Status 404 returned error can't find the container with id 5911af3ecd71a24bddc3bdbf4ca99965ec4bae25ec452edc8f99e96ac6acd360
Sep 17 14:06:09.750479 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:09.750450    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hm4vd"]
Sep 17 14:06:09.753738 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:09.753709    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c43836f_f444_4fa9_9155_b75f04b32d80.slice/crio-58e31d3e5e89dd20b2a415f90502124b5a21ae99266e903fc7bfed7811f7164f WatchSource:0}: Error finding container 58e31d3e5e89dd20b2a415f90502124b5a21ae99266e903fc7bfed7811f7164f: Status 404 returned error can't find the container with id 58e31d3e5e89dd20b2a415f90502124b5a21ae99266e903fc7bfed7811f7164f
Sep 17 14:06:10.210811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.210779    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lpgsd" event={"ID":"582bd8f2-b198-4a29-96d1-a838e1dad55c","Type":"ContainerStarted","Data":"7c5f18c393b1de4becf0853014b2014d46b30c8200497b4b3d9444482b249cc3"}
Sep 17 14:06:10.212110 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.212085    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hm4vd" event={"ID":"6c43836f-f444-4fa9-9155-b75f04b32d80","Type":"ContainerStarted","Data":"58e31d3e5e89dd20b2a415f90502124b5a21ae99266e903fc7bfed7811f7164f"}
Sep 17 14:06:10.213999 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.213965    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp" event={"ID":"c381f96e-b724-4232-a4ca-f06476d465a1","Type":"ContainerStarted","Data":"2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1"}
Sep 17 14:06:10.214114 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.214014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp" event={"ID":"c381f96e-b724-4232-a4ca-f06476d465a1","Type":"ContainerStarted","Data":"5911af3ecd71a24bddc3bdbf4ca99965ec4bae25ec452edc8f99e96ac6acd360"}
Sep 17 14:06:10.214423 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.214390    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:06:10.251546 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.251493    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp" podStartSLOduration=35.251473326 podStartE2EDuration="35.251473326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:10.248832923 +0000 UTC m=+67.840008700" watchObservedRunningTime="2026-09-17 14:06:10.251473326 +0000 UTC m=+67.842649087"
Sep 17 14:06:10.726132 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.725934    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"]
Sep 17 14:06:10.741401 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.741305    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"
Sep 17 14:06:10.766831 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.766732    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5sgz\" (UniqueName: \"kubernetes.io/projected/657a3314-4707-49bd-9b93-037f818fd746-kube-api-access-l5sgz\") pod \"managed-serviceaccount-addon-agent-7d79fc4766-92gbt\" (UID: \"657a3314-4707-49bd-9b93-037f818fd746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"
Sep 17 14:06:10.766831 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.766800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/657a3314-4707-49bd-9b93-037f818fd746-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d79fc4766-92gbt\" (UID: \"657a3314-4707-49bd-9b93-037f818fd746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"
Sep 17 14:06:10.779070 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.778346    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 14:06:10.779070 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.778460    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 14:06:10.779070 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.778700    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 14:06:10.779070 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.778890    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-j5zkz\""
Sep 17 14:06:10.780033 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.779843    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 14:06:10.780521 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.780238    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"]
Sep 17 14:06:10.801561 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.801526    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"]
Sep 17 14:06:10.801720 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.801658    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:10.805871 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.805654    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 14:06:10.814263 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.814202    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"]
Sep 17 14:06:10.825268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.825231    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65bb67c7d5-svssp"]
Sep 17 14:06:10.868310 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.868159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/657a3314-4707-49bd-9b93-037f818fd746-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d79fc4766-92gbt\" (UID: \"657a3314-4707-49bd-9b93-037f818fd746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"
Sep 17 14:06:10.868532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.868343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5sgz\" (UniqueName: \"kubernetes.io/projected/657a3314-4707-49bd-9b93-037f818fd746-kube-api-access-l5sgz\") pod \"managed-serviceaccount-addon-agent-7d79fc4766-92gbt\" (UID: \"657a3314-4707-49bd-9b93-037f818fd746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"
Sep 17 14:06:10.868532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.868393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e7f9decf-195d-4e03-9b3a-6cfe9590ec6a-klusterlet-config\") pod \"klusterlet-addon-workmgr-86965dfbdf-85dv9\" (UID: \"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:10.868532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.868420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7f9decf-195d-4e03-9b3a-6cfe9590ec6a-tmp\") pod \"klusterlet-addon-workmgr-86965dfbdf-85dv9\" (UID: \"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:10.868532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.868449    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn5h4\" (UniqueName: \"kubernetes.io/projected/e7f9decf-195d-4e03-9b3a-6cfe9590ec6a-kube-api-access-dn5h4\") pod \"klusterlet-addon-workmgr-86965dfbdf-85dv9\" (UID: \"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:10.874957 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.874897    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/657a3314-4707-49bd-9b93-037f818fd746-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d79fc4766-92gbt\" (UID: \"657a3314-4707-49bd-9b93-037f818fd746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"
Sep 17 14:06:10.897755 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.897692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5sgz\" (UniqueName: \"kubernetes.io/projected/657a3314-4707-49bd-9b93-037f818fd746-kube-api-access-l5sgz\") pod \"managed-serviceaccount-addon-agent-7d79fc4766-92gbt\" (UID: \"657a3314-4707-49bd-9b93-037f818fd746\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"
Sep 17 14:06:10.969585 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.969548    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e7f9decf-195d-4e03-9b3a-6cfe9590ec6a-klusterlet-config\") pod \"klusterlet-addon-workmgr-86965dfbdf-85dv9\" (UID: \"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:10.969776 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.969595    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7f9decf-195d-4e03-9b3a-6cfe9590ec6a-tmp\") pod \"klusterlet-addon-workmgr-86965dfbdf-85dv9\" (UID: \"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:10.969776 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.969631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn5h4\" (UniqueName: \"kubernetes.io/projected/e7f9decf-195d-4e03-9b3a-6cfe9590ec6a-kube-api-access-dn5h4\") pod \"klusterlet-addon-workmgr-86965dfbdf-85dv9\" (UID: \"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:10.970419 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.970389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7f9decf-195d-4e03-9b3a-6cfe9590ec6a-tmp\") pod \"klusterlet-addon-workmgr-86965dfbdf-85dv9\" (UID: \"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:10.973276 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:10.973248    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e7f9decf-195d-4e03-9b3a-6cfe9590ec6a-klusterlet-config\") pod \"klusterlet-addon-workmgr-86965dfbdf-85dv9\" (UID: \"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:11.020227 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.020145    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-544f7f87c9-dt7kb"]
Sep 17 14:06:11.031059 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.030997    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn5h4\" (UniqueName: \"kubernetes.io/projected/e7f9decf-195d-4e03-9b3a-6cfe9590ec6a-kube-api-access-dn5h4\") pod \"klusterlet-addon-workmgr-86965dfbdf-85dv9\" (UID: \"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:11.042756 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.042723    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.067697 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.067648    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"
Sep 17 14:06:11.070979 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.070924    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6690262-a99d-466e-bfbb-45d3e609d5ac-installation-pull-secrets\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.071113 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.070995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6690262-a99d-466e-bfbb-45d3e609d5ac-ca-trust-extracted\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.071113 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.071027    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqrlk\" (UniqueName: \"kubernetes.io/projected/f6690262-a99d-466e-bfbb-45d3e609d5ac-kube-api-access-wqrlk\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.071237 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.071126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6690262-a99d-466e-bfbb-45d3e609d5ac-bound-sa-token\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.071237 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.071153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6690262-a99d-466e-bfbb-45d3e609d5ac-registry-tls\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.071237 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.071189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6690262-a99d-466e-bfbb-45d3e609d5ac-image-registry-private-configuration\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.071400 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.071264    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6690262-a99d-466e-bfbb-45d3e609d5ac-trusted-ca\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.071400 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.071323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6690262-a99d-466e-bfbb-45d3e609d5ac-registry-certificates\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.114522 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.114490    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:11.121634 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.121551    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-544f7f87c9-dt7kb"]
Sep 17 14:06:11.172509 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.172349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6690262-a99d-466e-bfbb-45d3e609d5ac-bound-sa-token\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.172651 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.172524    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6690262-a99d-466e-bfbb-45d3e609d5ac-registry-tls\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.172651 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.172581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6690262-a99d-466e-bfbb-45d3e609d5ac-image-registry-private-configuration\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.172651 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.172611    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6690262-a99d-466e-bfbb-45d3e609d5ac-trusted-ca\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.172810 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.172659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6690262-a99d-466e-bfbb-45d3e609d5ac-registry-certificates\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.172810 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.172690    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6690262-a99d-466e-bfbb-45d3e609d5ac-installation-pull-secrets\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.172810 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.172719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6690262-a99d-466e-bfbb-45d3e609d5ac-ca-trust-extracted\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.172810 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.172751    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrlk\" (UniqueName: \"kubernetes.io/projected/f6690262-a99d-466e-bfbb-45d3e609d5ac-kube-api-access-wqrlk\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.174022 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.173975    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6690262-a99d-466e-bfbb-45d3e609d5ac-trusted-ca\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.174371 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.174324    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6690262-a99d-466e-bfbb-45d3e609d5ac-registry-certificates\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.174479 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.174408    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6690262-a99d-466e-bfbb-45d3e609d5ac-ca-trust-extracted\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.178969 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.178889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6690262-a99d-466e-bfbb-45d3e609d5ac-image-registry-private-configuration\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.181980 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.181955    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6690262-a99d-466e-bfbb-45d3e609d5ac-registry-tls\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.183086 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.183063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6690262-a99d-466e-bfbb-45d3e609d5ac-installation-pull-secrets\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.216014 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.215951    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrlk\" (UniqueName: \"kubernetes.io/projected/f6690262-a99d-466e-bfbb-45d3e609d5ac-kube-api-access-wqrlk\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.226630 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.226407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4bth" event={"ID":"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e","Type":"ContainerStarted","Data":"75625945afbc815d2764f183d15ac1437fa7494f29ec3af52de8e41c1c37bd3e"}
Sep 17 14:06:11.226630 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.226477    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4bth" event={"ID":"796b6e4c-79f6-4a2b-98a5-eb2f8daeb09e","Type":"ContainerStarted","Data":"1469986b11b1efb17251bb0b714bdd8a38c197e719622aa7dbabc754c3b71969"}
Sep 17 14:06:11.230112 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.230026    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6690262-a99d-466e-bfbb-45d3e609d5ac-bound-sa-token\") pod \"image-registry-544f7f87c9-dt7kb\" (UID: \"f6690262-a99d-466e-bfbb-45d3e609d5ac\") " pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.309837 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.309713    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m4bth" podStartSLOduration=66.883991307 podStartE2EDuration="1m8.309690316s" totalImagesPullingTime="1.425699009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:11.308818433 +0000 UTC m=+68.899994196" watchObservedRunningTime="2026-09-17 14:06:11.309690316 +0000 UTC m=+68.900866081"
Sep 17 14:06:11.356077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.355712    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:11.405645 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.405610    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt"]
Sep 17 14:06:11.474811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:11.474781    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"]
Sep 17 14:06:12.003993 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:12.003958    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f9decf_195d_4e03_9b3a_6cfe9590ec6a.slice/crio-4dbac35b9abf23c984f7de17d302d56d8099c85b966d4e9d220aec8e818bd3e4 WatchSource:0}: Error finding container 4dbac35b9abf23c984f7de17d302d56d8099c85b966d4e9d220aec8e818bd3e4: Status 404 returned error can't find the container with id 4dbac35b9abf23c984f7de17d302d56d8099c85b966d4e9d220aec8e818bd3e4
Sep 17 14:06:12.231709 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:12.231670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt" event={"ID":"657a3314-4707-49bd-9b93-037f818fd746","Type":"ContainerStarted","Data":"117c1d60dd21d6b4da7338a017c37e62e5103a04b1bedc9bc9a1ae3448b26447"}
Sep 17 14:06:12.233453 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:12.233421    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9" event={"ID":"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a","Type":"ContainerStarted","Data":"4dbac35b9abf23c984f7de17d302d56d8099c85b966d4e9d220aec8e818bd3e4"}
Sep 17 14:06:12.302446 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:12.302352    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-544f7f87c9-dt7kb"]
Sep 17 14:06:13.241848 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:13.241809    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hm4vd" event={"ID":"6c43836f-f444-4fa9-9155-b75f04b32d80","Type":"ContainerStarted","Data":"d60089f2e1e247e44e4df7e8f8af3b9a87d19efdc3d2f6c710eea48a2e39305b"}
Sep 17 14:06:14.633621 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:14.633581    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6690262_a99d_466e_bfbb_45d3e609d5ac.slice/crio-f39491d645aedddbacf4ea165d0b6ab3d3657bc5e118ef05dda2609dd3bbd733 WatchSource:0}: Error finding container f39491d645aedddbacf4ea165d0b6ab3d3657bc5e118ef05dda2609dd3bbd733: Status 404 returned error can't find the container with id f39491d645aedddbacf4ea165d0b6ab3d3657bc5e118ef05dda2609dd3bbd733
Sep 17 14:06:15.252683 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.252626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lpgsd" event={"ID":"582bd8f2-b198-4a29-96d1-a838e1dad55c","Type":"ContainerStarted","Data":"bd29385a32cca8b88276482f3a899b63f742ade58bb238647408795ca5864883"}
Sep 17 14:06:15.253431 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.253407    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:06:15.257071 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.257030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hm4vd" event={"ID":"6c43836f-f444-4fa9-9155-b75f04b32d80","Type":"ContainerStarted","Data":"5eca377d9de65292b1e356e7d4bff988f5a9b66f7ad64dd7c9c4113cc5f5c5d8"}
Sep 17 14:06:15.257659 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.257638    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:06:15.261414 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.260721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb" event={"ID":"f6690262-a99d-466e-bfbb-45d3e609d5ac","Type":"ContainerStarted","Data":"bb8d4bf2272f38d37528f1c421044f258eea607c3910e461e952021cfb2f570c"}
Sep 17 14:06:15.261414 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.260765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb" event={"ID":"f6690262-a99d-466e-bfbb-45d3e609d5ac","Type":"ContainerStarted","Data":"f39491d645aedddbacf4ea165d0b6ab3d3657bc5e118ef05dda2609dd3bbd733"}
Sep 17 14:06:15.261414 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.261347    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:15.263949 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.263927    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-52kgw" event={"ID":"3839cd65-b24b-4e79-b075-d703429c694f","Type":"ContainerStarted","Data":"4115d78792561277be4deb6825be9e5608ee010ed54cfd86481ed97e382b60ce"}
Sep 17 14:06:15.274536 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.274480    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lpgsd" podStartSLOduration=66.875852494 podStartE2EDuration="1m12.274460427s" totalImagesPullingTime="5.398607933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:15.273062656 +0000 UTC m=+72.864238418" watchObservedRunningTime="2026-09-17 14:06:15.274460427 +0000 UTC m=+72.865636190"
Sep 17 14:06:15.312649 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.312546    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb" podStartSLOduration=5.312523886 podStartE2EDuration="5.312523886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:15.310424138 +0000 UTC m=+72.901599902" watchObservedRunningTime="2026-09-17 14:06:15.312523886 +0000 UTC m=+72.903699651"
Sep 17 14:06:15.342441 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.342380    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-52kgw" podStartSLOduration=66.613871704 podStartE2EDuration="1m12.342342809s" totalImagesPullingTime="5.728471105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:15.339668068 +0000 UTC m=+72.930843832" watchObservedRunningTime="2026-09-17 14:06:15.342342809 +0000 UTC m=+72.933518572"
Sep 17 14:06:15.371830 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:15.371766    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hm4vd" podStartSLOduration=35.952440623 podStartE2EDuration="38.371743155s" totalImagesPullingTime="2.419302532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:15.368766772 +0000 UTC m=+72.959942546" watchObservedRunningTime="2026-09-17 14:06:15.371743155 +0000 UTC m=+72.962918917"
Sep 17 14:06:19.280955 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:19.280905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9" event={"ID":"e7f9decf-195d-4e03-9b3a-6cfe9590ec6a","Type":"ContainerStarted","Data":"c698c16d1c9e66a26250e1167a86816cfb2770b2d6bf2b3c34450f61565fedf0"}
Sep 17 14:06:19.281515 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:19.280982    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:19.282645 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:19.282615    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt" event={"ID":"657a3314-4707-49bd-9b93-037f818fd746","Type":"ContainerStarted","Data":"f7cdf27f9c3c0148310673e7c0c37a321d7c103d7efa6a54cad4d62dcf09d9fc"}
Sep 17 14:06:19.283013 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:19.282994    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9"
Sep 17 14:06:19.301929 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:19.301876    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86965dfbdf-85dv9" podStartSLOduration=2.515712366 podStartE2EDuration="9.301859354s" totalImagesPullingTime="6.786146988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:19.301528276 +0000 UTC m=+76.892704051" watchObservedRunningTime="2026-09-17 14:06:19.301859354 +0000 UTC m=+76.893035115"
Sep 17 14:06:19.326137 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:19.326079    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d79fc4766-92gbt" podStartSLOduration=2.579309596 podStartE2EDuration="9.326064059s" totalImagesPullingTime="6.746754463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:19.324865641 +0000 UTC m=+76.916041404" watchObservedRunningTime="2026-09-17 14:06:19.326064059 +0000 UTC m=+76.917239821"
Sep 17 14:06:23.273945 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.273913    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hm4vd"
Sep 17 14:06:23.312629 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.312599    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-txsnw"]
Sep 17 14:06:23.314890 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.314867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.321495 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.321477    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 14:06:23.321764 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.321744    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 14:06:23.321834 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.321785    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 14:06:23.321886 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.321829    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 14:06:23.322032 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.322007    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 14:06:23.325927 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.325912    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 14:06:23.326148 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.326136    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5jnvg\""
Sep 17 14:06:23.373155 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.373115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-accelerators-collector-config\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.373370 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.373179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-textfile\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.373370 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.373211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-tls\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.373370 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.373232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-wtmp\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.373370 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.373269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45svg\" (UniqueName: \"kubernetes.io/projected/442f751d-a8a9-43d2-ae0c-edaaa0092a52-kube-api-access-45svg\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.373370 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.373337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/442f751d-a8a9-43d2-ae0c-edaaa0092a52-root\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.373636 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.373415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.373636 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.373500    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/442f751d-a8a9-43d2-ae0c-edaaa0092a52-metrics-client-ca\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.373636 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.373532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442f751d-a8a9-43d2-ae0c-edaaa0092a52-sys\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.474876 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.474831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/442f751d-a8a9-43d2-ae0c-edaaa0092a52-metrics-client-ca\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.474876 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.474876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442f751d-a8a9-43d2-ae0c-edaaa0092a52-sys\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475133 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.474915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-accelerators-collector-config\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475133 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442f751d-a8a9-43d2-ae0c-edaaa0092a52-sys\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475133 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-textfile\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475133 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-tls\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475133 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-wtmp\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475419 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45svg\" (UniqueName: \"kubernetes.io/projected/442f751d-a8a9-43d2-ae0c-edaaa0092a52-kube-api-access-45svg\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475419 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/442f751d-a8a9-43d2-ae0c-edaaa0092a52-root\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475419 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475419 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:06:23.475223    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 14:06:23.475419 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:06:23.475301    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-tls podName:442f751d-a8a9-43d2-ae0c-edaaa0092a52 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:23.975279383 +0000 UTC m=+81.566455122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-tls") pod "node-exporter-txsnw" (UID: "442f751d-a8a9-43d2-ae0c-edaaa0092a52") : secret "node-exporter-tls" not found
Sep 17 14:06:23.475419 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-wtmp\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475725 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-textfile\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475725 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/442f751d-a8a9-43d2-ae0c-edaaa0092a52-root\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475725 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/442f751d-a8a9-43d2-ae0c-edaaa0092a52-metrics-client-ca\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.475725 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.475625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-accelerators-collector-config\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.477510 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.477494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.497066 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.497037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45svg\" (UniqueName: \"kubernetes.io/projected/442f751d-a8a9-43d2-ae0c-edaaa0092a52-kube-api-access-45svg\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.979657 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.979606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-tls\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:23.981922 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:23.981892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/442f751d-a8a9-43d2-ae0c-edaaa0092a52-node-exporter-tls\") pod \"node-exporter-txsnw\" (UID: \"442f751d-a8a9-43d2-ae0c-edaaa0092a52\") " pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:24.223606 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.223573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-txsnw"
Sep 17 14:06:24.233209 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:24.233183    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442f751d_a8a9_43d2_ae0c_edaaa0092a52.slice/crio-3c2d0043a33f983eef42cd6e88795d262acee94b3552e1be3eab37bed051ceb3 WatchSource:0}: Error finding container 3c2d0043a33f983eef42cd6e88795d262acee94b3552e1be3eab37bed051ceb3: Status 404 returned error can't find the container with id 3c2d0043a33f983eef42cd6e88795d262acee94b3552e1be3eab37bed051ceb3
Sep 17 14:06:24.298988 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.298954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txsnw" event={"ID":"442f751d-a8a9-43d2-ae0c-edaaa0092a52","Type":"ContainerStarted","Data":"3c2d0043a33f983eef42cd6e88795d262acee94b3552e1be3eab37bed051ceb3"}
Sep 17 14:06:24.409068 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.409036    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:06:24.412516 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.412493    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.421286 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.421266    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 14:06:24.423304 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.423288    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 14:06:24.423505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.423493    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g5tp7\""
Sep 17 14:06:24.423632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.423616    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 14:06:24.423812 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.423794    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 14:06:24.423873 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.423826    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 14:06:24.423928 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.423895    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 14:06:24.424474 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.424460    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 14:06:24.425054 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.425041    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 14:06:24.427073 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.427059    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 14:06:24.465840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.465812    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:06:24.482928 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.482860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.482928 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.482891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.482928 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.482911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.482928 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.482925    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.483135 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.482949    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xrj\" (UniqueName: \"kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-kube-api-access-g2xrj\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.483135 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.483014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.483135 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.483035    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.483135 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.483056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.483135 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.483072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.483135 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.483107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-web-config\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.483311 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.483144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.483311 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.483160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-out\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.483311 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.483187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584241 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584126    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584241 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584241 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584183    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xrj\" (UniqueName: \"kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-kube-api-access-g2xrj\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584241 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584241 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584625 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584625 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584625 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-web-config\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584625 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584625 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584403    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-out\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584884 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.584642    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.584884 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:06:24.584766    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-trusted-ca-bundle podName:0b02bdde-e086-4a5a-8f19-ecb6f9b05912 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:25.084744489 +0000 UTC m=+82.675920235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912") : configmap references non-existent config key: ca-bundle.crt
Sep 17 14:06:24.585033 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.585001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.585088 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.585062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.585139 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.585097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.585300 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.585210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.587723 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.587699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.587842 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.587707    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.588070 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.588048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-out\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.588151 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.588131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.588252 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.588223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.588252 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.588239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.588532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.588348    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.588532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.588498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-web-config\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.588944 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.588927    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:24.606208 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:24.606179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xrj\" (UniqueName: \"kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-kube-api-access-g2xrj\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:25.089780 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.089738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:25.090696 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.090660    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:25.304057 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.304021    2560 generic.go:356] "Generic (PLEG): container finished" podID="442f751d-a8a9-43d2-ae0c-edaaa0092a52" containerID="69e0e5d6eb16416ed82c9bbc6615c4a750079c273defd1d4784da7db028ea13c" exitCode=0
Sep 17 14:06:25.304416 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.304079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txsnw" event={"ID":"442f751d-a8a9-43d2-ae0c-edaaa0092a52","Type":"ContainerDied","Data":"69e0e5d6eb16416ed82c9bbc6615c4a750079c273defd1d4784da7db028ea13c"}
Sep 17 14:06:25.321832 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.321806    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:06:25.417208 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.417180    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"]
Sep 17 14:06:25.420241 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.420220    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.425470 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.425431    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 14:06:25.425627 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.425472    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vsd2n\""
Sep 17 14:06:25.426321 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.425888    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 14:06:25.426321 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.425900    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8rjl567hs4hu7\""
Sep 17 14:06:25.426321 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.426069    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 14:06:25.426321 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.426161    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 14:06:25.426321 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.426201    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 14:06:25.463732 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.463697    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"]
Sep 17 14:06:25.492262 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.492227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.492262 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.492260    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.492474 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.492282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-tls\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.492474 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.492307    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.492474 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.492341    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d3faec5-4509-4524-921b-cffb4fcae95b-metrics-client-ca\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.492474 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.492374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-grpc-tls\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.492474 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.492393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdhf\" (UniqueName: \"kubernetes.io/projected/4d3faec5-4509-4524-921b-cffb4fcae95b-kube-api-access-6gdhf\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.492474 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.492420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.593735 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.593700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.593899 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.593773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.593899 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.593804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.593899 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.593840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-tls\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.594821 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.594138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.594821 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.594204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d3faec5-4509-4524-921b-cffb4fcae95b-metrics-client-ca\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.594821 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.594244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-grpc-tls\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.594821 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.594276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdhf\" (UniqueName: \"kubernetes.io/projected/4d3faec5-4509-4524-921b-cffb4fcae95b-kube-api-access-6gdhf\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.595115 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.595007    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d3faec5-4509-4524-921b-cffb4fcae95b-metrics-client-ca\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.596737 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.596672    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-tls\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.596854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.596832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.596915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.596839    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.597025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.597006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-grpc-tls\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.597081 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.597044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.597135 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.597048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d3faec5-4509-4524-921b-cffb4fcae95b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.604737 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.604715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdhf\" (UniqueName: \"kubernetes.io/projected/4d3faec5-4509-4524-921b-cffb4fcae95b-kube-api-access-6gdhf\") pod \"thanos-querier-79fb5f4bb8-rhbgq\" (UID: \"4d3faec5-4509-4524-921b-cffb4fcae95b\") " pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:25.637110 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.637085    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:06:25.639658 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:25.639629    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b02bdde_e086_4a5a_8f19_ecb6f9b05912.slice/crio-dce519d2f997881eb7ee4ca034707b2e9c7aa93540accbf2809b687e7d22fd94 WatchSource:0}: Error finding container dce519d2f997881eb7ee4ca034707b2e9c7aa93540accbf2809b687e7d22fd94: Status 404 returned error can't find the container with id dce519d2f997881eb7ee4ca034707b2e9c7aa93540accbf2809b687e7d22fd94
Sep 17 14:06:25.729502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:25.729465    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:26.031684 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:26.031654    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"]
Sep 17 14:06:26.034389 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:26.034348    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3faec5_4509_4524_921b_cffb4fcae95b.slice/crio-aa7802a97e943fa2ffc68a91793c13ac30c163d954fcb52011c4a0b7a59d88f0 WatchSource:0}: Error finding container aa7802a97e943fa2ffc68a91793c13ac30c163d954fcb52011c4a0b7a59d88f0: Status 404 returned error can't find the container with id aa7802a97e943fa2ffc68a91793c13ac30c163d954fcb52011c4a0b7a59d88f0
Sep 17 14:06:26.311040 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:26.310947    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txsnw" event={"ID":"442f751d-a8a9-43d2-ae0c-edaaa0092a52","Type":"ContainerStarted","Data":"35640db4a1da334ca4dc22489fd72ab34a06a33ebe16ea00976a2704a39bc760"}
Sep 17 14:06:26.311040 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:26.311015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txsnw" event={"ID":"442f751d-a8a9-43d2-ae0c-edaaa0092a52","Type":"ContainerStarted","Data":"1e056490a0fc49516e21bcf03d0d36a68f0dc359314d718d17e856adf473338e"}
Sep 17 14:06:26.312493 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:26.312458    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq" event={"ID":"4d3faec5-4509-4524-921b-cffb4fcae95b","Type":"ContainerStarted","Data":"aa7802a97e943fa2ffc68a91793c13ac30c163d954fcb52011c4a0b7a59d88f0"}
Sep 17 14:06:26.313847 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:26.313818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerStarted","Data":"dce519d2f997881eb7ee4ca034707b2e9c7aa93540accbf2809b687e7d22fd94"}
Sep 17 14:06:26.339722 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:26.339667    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-txsnw" podStartSLOduration=2.368374271 podStartE2EDuration="3.339650934s" totalImagesPullingTime="909.67587ms" totalInitContainerRuntime="61.600793ms" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:26.339133596 +0000 UTC m=+83.930309357" watchObservedRunningTime="2026-09-17 14:06:26.339650934 +0000 UTC m=+83.930826737"
Sep 17 14:06:27.319742 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:27.319705    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerID="cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd" exitCode=0
Sep 17 14:06:27.320120 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:27.319792    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerDied","Data":"cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd"}
Sep 17 14:06:28.325446 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:28.325347    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq" event={"ID":"4d3faec5-4509-4524-921b-cffb4fcae95b","Type":"ContainerStarted","Data":"de2bc5e86ddd11bce162f876fdd4ef3d05655d320591a8c1a3f3ce04b5f82f0c"}
Sep 17 14:06:28.325446 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:28.325433    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq" event={"ID":"4d3faec5-4509-4524-921b-cffb4fcae95b","Type":"ContainerStarted","Data":"b6d7e30af94ae80b55cdc3878b5d487d47e8d397d6ff59b9d3e6940df39a7fff"}
Sep 17 14:06:28.325823 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:28.325459    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq" event={"ID":"4d3faec5-4509-4524-921b-cffb4fcae95b","Type":"ContainerStarted","Data":"5c1a344c36e85188d5ede7ba48428b3273e76d073eb050fa3e27169c9d761760"}
Sep 17 14:06:29.627694 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.627662    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:06:29.630902 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.630881    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.637012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.636988    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 14:06:29.637130 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.637086    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 14:06:29.638120 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.638097    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 14:06:29.638339 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.638320    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 14:06:29.638425 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.638380    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 14:06:29.638777 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.638761    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 14:06:29.638844 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.638761    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c7qa9d27b95cr\""
Sep 17 14:06:29.640677 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.640662    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 14:06:29.640860 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.640844    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 14:06:29.640860 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.640856    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 14:06:29.640991 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.640844    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 14:06:29.643772 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.643752    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jt6qk\""
Sep 17 14:06:29.643959 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.643946    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 14:06:29.644397 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.644381    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 14:06:29.644446 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.644415    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 14:06:29.644686 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.644672    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 14:06:29.648120 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.648100    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 14:06:29.655673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.655653    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:06:29.730980 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.730940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-config\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.730980 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.730976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731167 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.730993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731167 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-config-out\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731167 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731167 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731167 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731393 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731393 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731393 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731240    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zs8c\" (UniqueName: \"kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-kube-api-access-5zs8c\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731393 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731393 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731310    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731393 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731325    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731393 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731348    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.731618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.731499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-web-config\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.832667 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-config-out\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.832667 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.832813 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.832813 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.832883 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832814    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.832883 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.832883 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zs8c\" (UniqueName: \"kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-kube-api-access-5zs8c\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833027 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833027 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833027 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832955    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833027 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.832983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833027 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833072    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833104    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-web-config\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833213    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-config\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833259    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833739 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833485    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833800 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.833894 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.833870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.835688 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.835654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-config-out\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.836149 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.835830    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.836149 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.835862    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.836149 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.836003    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.837300 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.836579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.837300 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.836599    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-config\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.837300 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.836655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.837300 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.836705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.837300 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.836963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.837300 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.837185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.837300 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.837257    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.837715 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.837398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.838020 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.838001    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-web-config\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.838325 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.838297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.838605 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.838584    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.839351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.839332    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.842104 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.842087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zs8c\" (UniqueName: \"kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-kube-api-access-5zs8c\") pod \"prometheus-k8s-0\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:29.957603 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:29.957570    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:30.249840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.249803    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:06:30.254015 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:06:30.253990    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49c1100_de42_47b0_8370_a41c69b98353.slice/crio-aa2a34fd2d8c8aa2baaac95eedd091354aa19eee8fc6525dc2136c0e1689627c WatchSource:0}: Error finding container aa2a34fd2d8c8aa2baaac95eedd091354aa19eee8fc6525dc2136c0e1689627c: Status 404 returned error can't find the container with id aa2a34fd2d8c8aa2baaac95eedd091354aa19eee8fc6525dc2136c0e1689627c
Sep 17 14:06:30.336534 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.336500    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq" event={"ID":"4d3faec5-4509-4524-921b-cffb4fcae95b","Type":"ContainerStarted","Data":"86f45405f718fa7d75078f59a2143411006601c9872a5876397cc16e53e9f0d0"}
Sep 17 14:06:30.336718 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.336578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq" event={"ID":"4d3faec5-4509-4524-921b-cffb4fcae95b","Type":"ContainerStarted","Data":"09719ad975d27fbcc5736e6754808f15154d43e26807cd3a42d1801a0a10bba2"}
Sep 17 14:06:30.336718 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.336618    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq" event={"ID":"4d3faec5-4509-4524-921b-cffb4fcae95b","Type":"ContainerStarted","Data":"489500f720541652dc6890cb9a3655db37cfbc0fc664555a83c33115546cb9d9"}
Sep 17 14:06:30.336718 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.336661    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:30.340626 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.340591    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerStarted","Data":"b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5"}
Sep 17 14:06:30.340763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.340676    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerStarted","Data":"8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221"}
Sep 17 14:06:30.340763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.340720    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerStarted","Data":"7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13"}
Sep 17 14:06:30.340763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.340759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerStarted","Data":"02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1"}
Sep 17 14:06:30.340909 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.340789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerStarted","Data":"fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077"}
Sep 17 14:06:30.340909 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.340822    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerStarted","Data":"603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85"}
Sep 17 14:06:30.342072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.342048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerStarted","Data":"aa2a34fd2d8c8aa2baaac95eedd091354aa19eee8fc6525dc2136c0e1689627c"}
Sep 17 14:06:30.372768 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.372500    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq" podStartSLOduration=2.285885153 podStartE2EDuration="5.37247743s" totalImagesPullingTime="3.086592277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:30.371761888 +0000 UTC m=+87.962937652" watchObservedRunningTime="2026-09-17 14:06:30.37247743 +0000 UTC m=+87.963653193"
Sep 17 14:06:30.409552 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:30.409491    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.322764792 podStartE2EDuration="6.409472837s" totalImagesPullingTime="3.067440745s" totalInitContainerRuntime="19.2673ms" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:30.407165312 +0000 UTC m=+87.998341076" watchObservedRunningTime="2026-09-17 14:06:30.409472837 +0000 UTC m=+88.000648599"
Sep 17 14:06:31.348114 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:31.348071    2560 generic.go:356] "Generic (PLEG): container finished" podID="e49c1100-de42-47b0-8370-a41c69b98353" containerID="8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea" exitCode=0
Sep 17 14:06:31.348557 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:31.348152    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerDied","Data":"8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea"}
Sep 17 14:06:32.241495 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:32.241454    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:06:35.369484 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:35.369451    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerStarted","Data":"133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef"}
Sep 17 14:06:35.369930 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:35.369519    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerStarted","Data":"c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639"}
Sep 17 14:06:35.369930 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:35.369546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerStarted","Data":"236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea"}
Sep 17 14:06:35.369930 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:35.369570    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerStarted","Data":"6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7"}
Sep 17 14:06:35.369930 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:35.369594    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerStarted","Data":"e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807"}
Sep 17 14:06:35.369930 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:35.369619    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerStarted","Data":"70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac"}
Sep 17 14:06:35.419280 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:35.419219    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.121743349 podStartE2EDuration="6.419201242s" totalImagesPullingTime="3.277838144s" totalInitContainerRuntime="19.619749ms" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:06:35.415834242 +0000 UTC m=+93.007010025" watchObservedRunningTime="2026-09-17 14:06:35.419201242 +0000 UTC m=+93.010377004"
Sep 17 14:06:36.355465 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:36.355438    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79fb5f4bb8-rhbgq"
Sep 17 14:06:37.264430 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.264349    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp" podUID="c381f96e-b724-4232-a4ca-f06476d465a1" containerName="registry" containerID="cri-o://2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1" gracePeriod=30
Sep 17 14:06:37.276831 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.276807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-544f7f87c9-dt7kb"
Sep 17 14:06:37.505144 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.505119    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:06:37.602608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.602509    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-trusted-ca\") pod \"c381f96e-b724-4232-a4ca-f06476d465a1\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") "
Sep 17 14:06:37.602608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.602564    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-bound-sa-token\") pod \"c381f96e-b724-4232-a4ca-f06476d465a1\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") "
Sep 17 14:06:37.602608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.602585    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-registry-certificates\") pod \"c381f96e-b724-4232-a4ca-f06476d465a1\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") "
Sep 17 14:06:37.602608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.602609    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpts6\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-kube-api-access-xpts6\") pod \"c381f96e-b724-4232-a4ca-f06476d465a1\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") "
Sep 17 14:06:37.602932 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.602629    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") pod \"c381f96e-b724-4232-a4ca-f06476d465a1\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") "
Sep 17 14:06:37.602932 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.602693    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-image-registry-private-configuration\") pod \"c381f96e-b724-4232-a4ca-f06476d465a1\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") "
Sep 17 14:06:37.602932 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.602732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-installation-pull-secrets\") pod \"c381f96e-b724-4232-a4ca-f06476d465a1\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") "
Sep 17 14:06:37.602932 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.602755    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c381f96e-b724-4232-a4ca-f06476d465a1-ca-trust-extracted\") pod \"c381f96e-b724-4232-a4ca-f06476d465a1\" (UID: \"c381f96e-b724-4232-a4ca-f06476d465a1\") "
Sep 17 14:06:37.603229 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.603080    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c381f96e-b724-4232-a4ca-f06476d465a1" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:06:37.603229 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.603098    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c381f96e-b724-4232-a4ca-f06476d465a1" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:06:37.605469 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.605437    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c381f96e-b724-4232-a4ca-f06476d465a1" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:06:37.605612 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.605580    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c381f96e-b724-4232-a4ca-f06476d465a1" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:06:37.605703 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.605626    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-kube-api-access-xpts6" (OuterVolumeSpecName: "kube-api-access-xpts6") pod "c381f96e-b724-4232-a4ca-f06476d465a1" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1"). InnerVolumeSpecName "kube-api-access-xpts6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:06:37.605703 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.605634    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c381f96e-b724-4232-a4ca-f06476d465a1" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:06:37.605703 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.605678    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c381f96e-b724-4232-a4ca-f06476d465a1" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:06:37.613299 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.613267    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c381f96e-b724-4232-a4ca-f06476d465a1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c381f96e-b724-4232-a4ca-f06476d465a1" (UID: "c381f96e-b724-4232-a4ca-f06476d465a1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:06:37.704194 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.704150    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-image-registry-private-configuration\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:06:37.704194 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.704186    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c381f96e-b724-4232-a4ca-f06476d465a1-installation-pull-secrets\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:06:37.704194 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.704196    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c381f96e-b724-4232-a4ca-f06476d465a1-ca-trust-extracted\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:06:37.704194 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.704207    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-trusted-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:06:37.704498 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.704216    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-bound-sa-token\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:06:37.704498 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.704224    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c381f96e-b724-4232-a4ca-f06476d465a1-registry-certificates\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:06:37.704498 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.704232    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpts6\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-kube-api-access-xpts6\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:06:37.704498 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:37.704240    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c381f96e-b724-4232-a4ca-f06476d465a1-registry-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:06:38.382223 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:38.382189    2560 generic.go:356] "Generic (PLEG): container finished" podID="c381f96e-b724-4232-a4ca-f06476d465a1" containerID="2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1" exitCode=0
Sep 17 14:06:38.382704 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:38.382236    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp"
Sep 17 14:06:38.382704 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:38.382248    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp" event={"ID":"c381f96e-b724-4232-a4ca-f06476d465a1","Type":"ContainerDied","Data":"2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1"}
Sep 17 14:06:38.382704 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:38.382289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65bb67c7d5-svssp" event={"ID":"c381f96e-b724-4232-a4ca-f06476d465a1","Type":"ContainerDied","Data":"5911af3ecd71a24bddc3bdbf4ca99965ec4bae25ec452edc8f99e96ac6acd360"}
Sep 17 14:06:38.382704 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:38.382310    2560 scope.go:117] "RemoveContainer" containerID="2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1"
Sep 17 14:06:38.391735 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:38.391715    2560 scope.go:117] "RemoveContainer" containerID="2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1"
Sep 17 14:06:38.392036 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:06:38.392014    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1\": container with ID starting with 2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1 not found: ID does not exist" containerID="2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1"
Sep 17 14:06:38.392110 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:38.392046    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1"} err="failed to get container status \"2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1\": rpc error: code = NotFound desc = could not find container \"2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1\": container with ID starting with 2b1dc60b669517663dc0d12635df4194a749fe206130e2cecaf0802a7436d6c1 not found: ID does not exist"
Sep 17 14:06:38.405011 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:38.404981    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65bb67c7d5-svssp"]
Sep 17 14:06:38.410438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:38.410410    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65bb67c7d5-svssp"]
Sep 17 14:06:38.964406 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:38.964348    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c381f96e-b724-4232-a4ca-f06476d465a1" path="/var/lib/kubelet/pods/c381f96e-b724-4232-a4ca-f06476d465a1/volumes"
Sep 17 14:06:39.958708 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:39.958664    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:06:47.274504 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:06:47.274472    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lpgsd"
Sep 17 14:07:29.958227 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:29.958176    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:29.977885 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:29.977857    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:30.594068 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:30.594035    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:43.709419 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:43.709384    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:07:43.709854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:43.709666    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="alertmanager" containerID="cri-o://603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85" gracePeriod=120
Sep 17 14:07:43.709854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:43.709755    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="prom-label-proxy" containerID="cri-o://b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5" gracePeriod=120
Sep 17 14:07:43.709854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:43.709761    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy-metric" containerID="cri-o://8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221" gracePeriod=120
Sep 17 14:07:43.709854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:43.709804    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="config-reloader" containerID="cri-o://fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077" gracePeriod=120
Sep 17 14:07:43.709854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:43.709851    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy" containerID="cri-o://7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13" gracePeriod=120
Sep 17 14:07:43.710098 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:43.709809    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy-web" containerID="cri-o://02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1" gracePeriod=120
Sep 17 14:07:44.636630 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:44.636593    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerID="b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5" exitCode=0
Sep 17 14:07:44.636630 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:44.636618    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerID="7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13" exitCode=0
Sep 17 14:07:44.636630 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:44.636625    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerID="fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077" exitCode=0
Sep 17 14:07:44.636630 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:44.636631    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerID="603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85" exitCode=0
Sep 17 14:07:44.636893 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:44.636663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerDied","Data":"b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5"}
Sep 17 14:07:44.636893 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:44.636733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerDied","Data":"7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13"}
Sep 17 14:07:44.636893 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:44.636762    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerDied","Data":"fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077"}
Sep 17 14:07:44.636893 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:44.636787    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerDied","Data":"603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85"}
Sep 17 14:07:44.961434 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:44.961401    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.081443 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081401    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2xrj\" (UniqueName: \"kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-kube-api-access-g2xrj\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081443 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081448    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-cluster-tls-config\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081467    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-metrics-client-ca\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081491    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081519    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-web-config\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081553    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-web\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081570    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-trusted-ca-bundle\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081593    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-main-tls\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081619    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-main-db\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081655    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-volume\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081678    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-tls-assets\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.081705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081711    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-out\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.082185 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.081761    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy\") pod \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\" (UID: \"0b02bdde-e086-4a5a-8f19-ecb6f9b05912\") "
Sep 17 14:07:45.082185 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.082025    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:07:45.084575 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.084533    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:45.084802 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.084769    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:45.084875 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.084786    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:45.084875 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.084845    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:07:45.085034 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.085016    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:07:45.085316 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.085239    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-kube-api-access-g2xrj" (OuterVolumeSpecName: "kube-api-access-g2xrj") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "kube-api-access-g2xrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:07:45.085562 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.085528    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-volume" (OuterVolumeSpecName: "config-volume") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:45.085562 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.085542    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:07:45.086844 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.086818    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-out" (OuterVolumeSpecName: "config-out") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:07:45.087017 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.086996    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:45.088476 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.088440    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:45.094269 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.094243    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-web-config" (OuterVolumeSpecName: "web-config") pod "0b02bdde-e086-4a5a-8f19-ecb6f9b05912" (UID: "0b02bdde-e086-4a5a-8f19-ecb6f9b05912"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:45.182915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182873    2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-cluster-tls-config\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.182915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182906    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-metrics-client-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.182915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182918    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.182915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182928    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-web-config\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.183166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182938    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.183166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182948    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.183166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182958    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-main-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.183166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182966    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-alertmanager-main-db\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.183166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182976    2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.183166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182984    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-tls-assets\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.183166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182991    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-config-out\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.183166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.182998    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.183166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.183007    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2xrj\" (UniqueName: \"kubernetes.io/projected/0b02bdde-e086-4a5a-8f19-ecb6f9b05912-kube-api-access-g2xrj\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:45.643608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.643573    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerID="8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221" exitCode=0
Sep 17 14:07:45.643608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.643599    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerID="02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1" exitCode=0
Sep 17 14:07:45.643810 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.643635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerDied","Data":"8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221"}
Sep 17 14:07:45.643810 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.643661    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.643810 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.643738    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerDied","Data":"02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1"}
Sep 17 14:07:45.643810 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.643768    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b02bdde-e086-4a5a-8f19-ecb6f9b05912","Type":"ContainerDied","Data":"dce519d2f997881eb7ee4ca034707b2e9c7aa93540accbf2809b687e7d22fd94"}
Sep 17 14:07:45.643810 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.643799    2560 scope.go:117] "RemoveContainer" containerID="b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5"
Sep 17 14:07:45.652986 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.652966    2560 scope.go:117] "RemoveContainer" containerID="8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221"
Sep 17 14:07:45.660938 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.660913    2560 scope.go:117] "RemoveContainer" containerID="7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13"
Sep 17 14:07:45.668085 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.668059    2560 scope.go:117] "RemoveContainer" containerID="02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1"
Sep 17 14:07:45.673967 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.673940    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:07:45.676062 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.676045    2560 scope.go:117] "RemoveContainer" containerID="fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077"
Sep 17 14:07:45.684324 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.684300    2560 scope.go:117] "RemoveContainer" containerID="603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85"
Sep 17 14:07:45.684705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.684681    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:07:45.691819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.691799    2560 scope.go:117] "RemoveContainer" containerID="cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd"
Sep 17 14:07:45.699233 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.699211    2560 scope.go:117] "RemoveContainer" containerID="b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5"
Sep 17 14:07:45.699565 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:45.699546    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5\": container with ID starting with b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5 not found: ID does not exist" containerID="b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5"
Sep 17 14:07:45.699630 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.699575    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5"} err="failed to get container status \"b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5\": rpc error: code = NotFound desc = could not find container \"b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5\": container with ID starting with b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5 not found: ID does not exist"
Sep 17 14:07:45.699630 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.699597    2560 scope.go:117] "RemoveContainer" containerID="8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221"
Sep 17 14:07:45.699856 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:45.699842    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221\": container with ID starting with 8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221 not found: ID does not exist" containerID="8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221"
Sep 17 14:07:45.699897 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.699858    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221"} err="failed to get container status \"8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221\": rpc error: code = NotFound desc = could not find container \"8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221\": container with ID starting with 8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221 not found: ID does not exist"
Sep 17 14:07:45.699897 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.699870    2560 scope.go:117] "RemoveContainer" containerID="7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13"
Sep 17 14:07:45.700100 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:45.700084    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13\": container with ID starting with 7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13 not found: ID does not exist" containerID="7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13"
Sep 17 14:07:45.700139 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.700104    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13"} err="failed to get container status \"7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13\": rpc error: code = NotFound desc = could not find container \"7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13\": container with ID starting with 7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13 not found: ID does not exist"
Sep 17 14:07:45.700139 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.700118    2560 scope.go:117] "RemoveContainer" containerID="02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1"
Sep 17 14:07:45.700321 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:45.700305    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1\": container with ID starting with 02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1 not found: ID does not exist" containerID="02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1"
Sep 17 14:07:45.700394 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.700324    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1"} err="failed to get container status \"02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1\": rpc error: code = NotFound desc = could not find container \"02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1\": container with ID starting with 02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1 not found: ID does not exist"
Sep 17 14:07:45.700394 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.700337    2560 scope.go:117] "RemoveContainer" containerID="fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077"
Sep 17 14:07:45.700620 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:45.700604    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077\": container with ID starting with fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077 not found: ID does not exist" containerID="fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077"
Sep 17 14:07:45.700661 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.700625    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077"} err="failed to get container status \"fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077\": rpc error: code = NotFound desc = could not find container \"fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077\": container with ID starting with fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077 not found: ID does not exist"
Sep 17 14:07:45.700661 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.700639    2560 scope.go:117] "RemoveContainer" containerID="603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85"
Sep 17 14:07:45.700836 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:45.700821    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85\": container with ID starting with 603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85 not found: ID does not exist" containerID="603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85"
Sep 17 14:07:45.700875 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.700839    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85"} err="failed to get container status \"603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85\": rpc error: code = NotFound desc = could not find container \"603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85\": container with ID starting with 603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85 not found: ID does not exist"
Sep 17 14:07:45.700875 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.700852    2560 scope.go:117] "RemoveContainer" containerID="cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd"
Sep 17 14:07:45.701049 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:45.701034    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd\": container with ID starting with cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd not found: ID does not exist" containerID="cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd"
Sep 17 14:07:45.701093 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.701051    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd"} err="failed to get container status \"cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd\": rpc error: code = NotFound desc = could not find container \"cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd\": container with ID starting with cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd not found: ID does not exist"
Sep 17 14:07:45.701093 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.701063    2560 scope.go:117] "RemoveContainer" containerID="b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5"
Sep 17 14:07:45.701254 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.701239    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5"} err="failed to get container status \"b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5\": rpc error: code = NotFound desc = could not find container \"b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5\": container with ID starting with b27fd237088ee93fb1f2229987fd97359ca8ad4cd92553c1135e2417db53d8e5 not found: ID does not exist"
Sep 17 14:07:45.701295 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.701254    2560 scope.go:117] "RemoveContainer" containerID="8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221"
Sep 17 14:07:45.701492 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.701475    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221"} err="failed to get container status \"8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221\": rpc error: code = NotFound desc = could not find container \"8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221\": container with ID starting with 8c9579b624e4ce0932741aa90395e8e30218b0fda4c345f6147cc63ba1c76221 not found: ID does not exist"
Sep 17 14:07:45.701541 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.701492    2560 scope.go:117] "RemoveContainer" containerID="7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13"
Sep 17 14:07:45.701705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.701686    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13"} err="failed to get container status \"7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13\": rpc error: code = NotFound desc = could not find container \"7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13\": container with ID starting with 7ed339c5f34b57d1e14694dd0bde058d1cba6009c5b87db25381838d9276bb13 not found: ID does not exist"
Sep 17 14:07:45.701752 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.701707    2560 scope.go:117] "RemoveContainer" containerID="02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1"
Sep 17 14:07:45.701904 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.701886    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1"} err="failed to get container status \"02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1\": rpc error: code = NotFound desc = could not find container \"02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1\": container with ID starting with 02ee6df0e246f3ad1d441fdb9d8be104303a492be558b44c6654e7da49a0a2b1 not found: ID does not exist"
Sep 17 14:07:45.701973 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.701917    2560 scope.go:117] "RemoveContainer" containerID="fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077"
Sep 17 14:07:45.702171 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.702155    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077"} err="failed to get container status \"fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077\": rpc error: code = NotFound desc = could not find container \"fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077\": container with ID starting with fa572aec52f2438693c3d49e049b41bdc6ef30bbabfb371abb6ceb65b712b077 not found: ID does not exist"
Sep 17 14:07:45.702217 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.702171    2560 scope.go:117] "RemoveContainer" containerID="603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85"
Sep 17 14:07:45.702380 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.702347    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85"} err="failed to get container status \"603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85\": rpc error: code = NotFound desc = could not find container \"603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85\": container with ID starting with 603878a4e10e556812c887fc1da1f7e2a13810e8250339fb0b0023a0a5ce4f85 not found: ID does not exist"
Sep 17 14:07:45.702420 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.702381    2560 scope.go:117] "RemoveContainer" containerID="cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd"
Sep 17 14:07:45.702598 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.702581    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd"} err="failed to get container status \"cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd\": rpc error: code = NotFound desc = could not find container \"cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd\": container with ID starting with cec4cdc21eb3ecffc87034bcdda3218a3a4010c7958de82cf852de656ea668cd not found: ID does not exist"
Sep 17 14:07:45.720851 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.720819    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:07:45.721104 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721088    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="config-reloader"
Sep 17 14:07:45.721157 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721106    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="config-reloader"
Sep 17 14:07:45.721157 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721118    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c381f96e-b724-4232-a4ca-f06476d465a1" containerName="registry"
Sep 17 14:07:45.721157 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721125    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c381f96e-b724-4232-a4ca-f06476d465a1" containerName="registry"
Sep 17 14:07:45.721157 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721134    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="prom-label-proxy"
Sep 17 14:07:45.721157 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721140    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="prom-label-proxy"
Sep 17 14:07:45.721157 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721149    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="alertmanager"
Sep 17 14:07:45.721157 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721154    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="alertmanager"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721161    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy-metric"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721166    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy-metric"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721174    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy-web"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721180    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy-web"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721186    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="init-config-reloader"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721191    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="init-config-reloader"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721197    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721202    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721246    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy-web"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721253    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="config-reloader"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721261    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c381f96e-b724-4232-a4ca-f06476d465a1" containerName="registry"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721267    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721273    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="kube-rbac-proxy-metric"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721279    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="alertmanager"
Sep 17 14:07:45.721351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.721284    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" containerName="prom-label-proxy"
Sep 17 14:07:45.724851 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.724831    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.729819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.729795    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 14:07:45.729951 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.729898    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 14:07:45.730337 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.730170    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 14:07:45.730337 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.730184    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 14:07:45.730337 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.730255    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 14:07:45.730585 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.730567    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 14:07:45.730930 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.730913    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 14:07:45.731008 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.730981    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g5tp7\""
Sep 17 14:07:45.731645 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.731625    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 14:07:45.739811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.739783    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 14:07:45.746763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.746735    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:07:45.787644 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.787777 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.787777 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.787777 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787745    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b610e21-46df-49f0-bc86-2cbd5691d2ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.787891 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b610e21-46df-49f0-bc86-2cbd5691d2ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.787891 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b610e21-46df-49f0-bc86-2cbd5691d2ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.787891 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.787891 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.788008 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b610e21-46df-49f0-bc86-2cbd5691d2ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.788008 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-web-config\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.788008 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.787989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.788008 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.788003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b610e21-46df-49f0-bc86-2cbd5691d2ee-config-out\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.788135 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.788033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5xx\" (UniqueName: \"kubernetes.io/projected/1b610e21-46df-49f0-bc86-2cbd5691d2ee-kube-api-access-pz5xx\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.888843 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.888803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889050 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.888852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889050 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.888879    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b610e21-46df-49f0-bc86-2cbd5691d2ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889050 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.888904    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b610e21-46df-49f0-bc86-2cbd5691d2ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889050 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.888936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b610e21-46df-49f0-bc86-2cbd5691d2ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889050 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.888962    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889291 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.889103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889291 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.889184    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b610e21-46df-49f0-bc86-2cbd5691d2ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889291 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.889218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-web-config\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889291 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.889247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889291 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.889285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b610e21-46df-49f0-bc86-2cbd5691d2ee-config-out\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889525 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.889321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5xx\" (UniqueName: \"kubernetes.io/projected/1b610e21-46df-49f0-bc86-2cbd5691d2ee-kube-api-access-pz5xx\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889525 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.889324    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b610e21-46df-49f0-bc86-2cbd5691d2ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.889631 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.889613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.890407 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.890084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b610e21-46df-49f0-bc86-2cbd5691d2ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.890613 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.890591    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b610e21-46df-49f0-bc86-2cbd5691d2ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.892454 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.892423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b610e21-46df-49f0-bc86-2cbd5691d2ee-config-out\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.892600 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.892584    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.892672 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.892634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b610e21-46df-49f0-bc86-2cbd5691d2ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.892953 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.892927    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.893026 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.892946    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.893026 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.892968    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.893494 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.893471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.893903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.893866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.893903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.893873    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b610e21-46df-49f0-bc86-2cbd5691d2ee-web-config\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:45.900350 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:45.900320    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5xx\" (UniqueName: \"kubernetes.io/projected/1b610e21-46df-49f0-bc86-2cbd5691d2ee-kube-api-access-pz5xx\") pod \"alertmanager-main-0\" (UID: \"1b610e21-46df-49f0-bc86-2cbd5691d2ee\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:46.033905 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:46.033866    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:07:46.322693 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:46.322592    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:07:46.325268 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:07:46.325240    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b610e21_46df_49f0_bc86_2cbd5691d2ee.slice/crio-65a742453ec2f8f026121469efdc13396f16269b2cf19e9c054a11d22e6f1fcb WatchSource:0}: Error finding container 65a742453ec2f8f026121469efdc13396f16269b2cf19e9c054a11d22e6f1fcb: Status 404 returned error can't find the container with id 65a742453ec2f8f026121469efdc13396f16269b2cf19e9c054a11d22e6f1fcb
Sep 17 14:07:46.649387 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:46.649270    2560 generic.go:356] "Generic (PLEG): container finished" podID="1b610e21-46df-49f0-bc86-2cbd5691d2ee" containerID="ccaf568e40262a0205bff9d6cf0bd14a1df96982c5a5fe5299c3372ec1c0128f" exitCode=0
Sep 17 14:07:46.649558 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:46.649385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b610e21-46df-49f0-bc86-2cbd5691d2ee","Type":"ContainerDied","Data":"ccaf568e40262a0205bff9d6cf0bd14a1df96982c5a5fe5299c3372ec1c0128f"}
Sep 17 14:07:46.649558 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:46.649475    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b610e21-46df-49f0-bc86-2cbd5691d2ee","Type":"ContainerStarted","Data":"65a742453ec2f8f026121469efdc13396f16269b2cf19e9c054a11d22e6f1fcb"}
Sep 17 14:07:46.965339 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:46.965312    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b02bdde-e086-4a5a-8f19-ecb6f9b05912" path="/var/lib/kubelet/pods/0b02bdde-e086-4a5a-8f19-ecb6f9b05912/volumes"
Sep 17 14:07:47.658650 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:47.658614    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b610e21-46df-49f0-bc86-2cbd5691d2ee","Type":"ContainerStarted","Data":"cd451d65f295ea1c70d672ee3e65409372d0285329644fa2ffe4b920a1d8634b"}
Sep 17 14:07:47.659031 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:47.658686    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b610e21-46df-49f0-bc86-2cbd5691d2ee","Type":"ContainerStarted","Data":"cfab87f3cb8bcad6c722ef6eb0defec2b913cb67aa30cd7929c494c6c8dca64b"}
Sep 17 14:07:47.659031 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:47.658716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b610e21-46df-49f0-bc86-2cbd5691d2ee","Type":"ContainerStarted","Data":"bf55adbc0d940716f0966b12b53a075696970b401764b0b8c20ec7e56cf5c475"}
Sep 17 14:07:47.659031 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:47.658752    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b610e21-46df-49f0-bc86-2cbd5691d2ee","Type":"ContainerStarted","Data":"579216bb51be472ed18ac4792eb02c28818ed7b23e85dd665ec72720a661b385"}
Sep 17 14:07:47.659031 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:47.658777    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b610e21-46df-49f0-bc86-2cbd5691d2ee","Type":"ContainerStarted","Data":"cfdd00b80a945c2ce9c38a75f86787f63887fb48d93fa762240de401e42ed5ee"}
Sep 17 14:07:47.659031 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:47.658799    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b610e21-46df-49f0-bc86-2cbd5691d2ee","Type":"ContainerStarted","Data":"8975c498238328528073cc9a6b4b4705e6acd74dc68af96fda3ce0390124c6db"}
Sep 17 14:07:47.709172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:47.709110    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.691854712 podStartE2EDuration="2.709092443s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.237731ms" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:47.707012379 +0000 UTC m=+165.298188142" watchObservedRunningTime="2026-09-17 14:07:47.709092443 +0000 UTC m=+165.300268230"
Sep 17 14:07:48.088271 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.088229    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:07:48.088647 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.088605    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="prometheus" containerID="cri-o://70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac" gracePeriod=600
Sep 17 14:07:48.088773 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.088640    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy-thanos" containerID="cri-o://133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef" gracePeriod=600
Sep 17 14:07:48.088773 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.088678    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="config-reloader" containerID="cri-o://e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807" gracePeriod=600
Sep 17 14:07:48.088773 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.088616    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy" containerID="cri-o://c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639" gracePeriod=600
Sep 17 14:07:48.088773 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.088646    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="thanos-sidecar" containerID="cri-o://6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7" gracePeriod=600
Sep 17 14:07:48.088966 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.088647    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy-web" containerID="cri-o://236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea" gracePeriod=600
Sep 17 14:07:48.668606 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.668572    2560 generic.go:356] "Generic (PLEG): container finished" podID="e49c1100-de42-47b0-8370-a41c69b98353" containerID="133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef" exitCode=0
Sep 17 14:07:48.668606 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.668599    2560 generic.go:356] "Generic (PLEG): container finished" podID="e49c1100-de42-47b0-8370-a41c69b98353" containerID="c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639" exitCode=0
Sep 17 14:07:48.668606 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.668607    2560 generic.go:356] "Generic (PLEG): container finished" podID="e49c1100-de42-47b0-8370-a41c69b98353" containerID="6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7" exitCode=0
Sep 17 14:07:48.669159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.668616    2560 generic.go:356] "Generic (PLEG): container finished" podID="e49c1100-de42-47b0-8370-a41c69b98353" containerID="e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807" exitCode=0
Sep 17 14:07:48.669159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.668624    2560 generic.go:356] "Generic (PLEG): container finished" podID="e49c1100-de42-47b0-8370-a41c69b98353" containerID="70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac" exitCode=0
Sep 17 14:07:48.669159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.668647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerDied","Data":"133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef"}
Sep 17 14:07:48.669159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.668718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerDied","Data":"c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639"}
Sep 17 14:07:48.669159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.668746    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerDied","Data":"6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7"}
Sep 17 14:07:48.669159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.668770    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerDied","Data":"e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807"}
Sep 17 14:07:48.669159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:48.668793    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerDied","Data":"70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac"}
Sep 17 14:07:49.334899 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.334875    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.420055 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420011    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-config-out\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420055 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420048    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-metrics-client-ca\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420283 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420066    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-serving-certs-ca-bundle\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420283 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420091    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zs8c\" (UniqueName: \"kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-kube-api-access-5zs8c\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420283 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420112    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-db\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420283 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420235    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-kubelet-serving-ca-bundle\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420283 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420280    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-tls-assets\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-metrics-client-certs\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420336    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-0\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420383    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-config\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420413    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-kube-rbac-proxy\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420448    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420478    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-2\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420507    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-web-config\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420515    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:07:49.420547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420539    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-thanos-prometheus-http-client-file\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420960 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420567    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:07:49.420960 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420599    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-1\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420960 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420647    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420960 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420678    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-trusted-ca-bundle\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420960 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420701    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-tls\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.420960 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.420731    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-grpc-tls\") pod \"e49c1100-de42-47b0-8370-a41c69b98353\" (UID: \"e49c1100-de42-47b0-8370-a41c69b98353\") "
Sep 17 14:07:49.421574 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.421003    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-metrics-client-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.421574 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.421022    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.421574 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.421269    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:07:49.421803 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.421775    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:07:49.422329 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.422301    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:07:49.422329 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.422318    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:07:49.422953 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.422677    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:07:49.423724 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.423424    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:07:49.423724 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.423686    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:49.425106 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.425074    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:07:49.425210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.425188    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:49.425461 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.425437    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-config-out" (OuterVolumeSpecName: "config-out") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:07:49.425970 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.425946    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-kube-api-access-5zs8c" (OuterVolumeSpecName: "kube-api-access-5zs8c") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "kube-api-access-5zs8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:07:49.426461 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.426147    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:49.426547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.426482    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:49.426612 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.426561    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-config" (OuterVolumeSpecName: "config") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:49.426612 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.426601    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:49.426700 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.426670    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:49.426866 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.426838    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:49.436908 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.436876    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-web-config" (OuterVolumeSpecName: "web-config") pod "e49c1100-de42-47b0-8370-a41c69b98353" (UID: "e49c1100-de42-47b0-8370-a41c69b98353"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:07:49.522428 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522350    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522428 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522421    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522428 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522433    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522428 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522442    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-grpc-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522428 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522452    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-config-out\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522460    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zs8c\" (UniqueName: \"kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-kube-api-access-5zs8c\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522468    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-db\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522477    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522485    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49c1100-de42-47b0-8370-a41c69b98353-tls-assets\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522493    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-metrics-client-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522501    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522509    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-config\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522518    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-kube-rbac-proxy\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522526    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522534    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522546    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-web-config\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522554    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49c1100-de42-47b0-8370-a41c69b98353-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.522729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.522562    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49c1100-de42-47b0-8370-a41c69b98353-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:07:49.681140 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.681046    2560 generic.go:356] "Generic (PLEG): container finished" podID="e49c1100-de42-47b0-8370-a41c69b98353" containerID="236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea" exitCode=0
Sep 17 14:07:49.681140 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.681114    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerDied","Data":"236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea"}
Sep 17 14:07:49.681140 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.681133    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.681644 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.681205    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49c1100-de42-47b0-8370-a41c69b98353","Type":"ContainerDied","Data":"aa2a34fd2d8c8aa2baaac95eedd091354aa19eee8fc6525dc2136c0e1689627c"}
Sep 17 14:07:49.681644 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.681241    2560 scope.go:117] "RemoveContainer" containerID="133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef"
Sep 17 14:07:49.689987 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.689965    2560 scope.go:117] "RemoveContainer" containerID="c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639"
Sep 17 14:07:49.697494 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.697473    2560 scope.go:117] "RemoveContainer" containerID="236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea"
Sep 17 14:07:49.704704 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.704686    2560 scope.go:117] "RemoveContainer" containerID="6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7"
Sep 17 14:07:49.708413 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.708388    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:07:49.712087 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.712066    2560 scope.go:117] "RemoveContainer" containerID="e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807"
Sep 17 14:07:49.719503 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.719478    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:07:49.719932 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.719916    2560 scope.go:117] "RemoveContainer" containerID="70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac"
Sep 17 14:07:49.727593 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.727571    2560 scope.go:117] "RemoveContainer" containerID="8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea"
Sep 17 14:07:49.734879 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.734859    2560 scope.go:117] "RemoveContainer" containerID="133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef"
Sep 17 14:07:49.735197 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:49.735178    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef\": container with ID starting with 133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef not found: ID does not exist" containerID="133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef"
Sep 17 14:07:49.735269 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.735210    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef"} err="failed to get container status \"133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef\": rpc error: code = NotFound desc = could not find container \"133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef\": container with ID starting with 133367d0a0c58efc90b1b517a4e35c15a41e824563ee03f8f49254451d5e34ef not found: ID does not exist"
Sep 17 14:07:49.735269 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.735240    2560 scope.go:117] "RemoveContainer" containerID="c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639"
Sep 17 14:07:49.735541 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:49.735522    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639\": container with ID starting with c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639 not found: ID does not exist" containerID="c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639"
Sep 17 14:07:49.735578 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.735549    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639"} err="failed to get container status \"c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639\": rpc error: code = NotFound desc = could not find container \"c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639\": container with ID starting with c31fb8564289ba5aad7e78fc68fbb3e2f531b67e3c28231fdbc5a5c29e168639 not found: ID does not exist"
Sep 17 14:07:49.735578 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.735565    2560 scope.go:117] "RemoveContainer" containerID="236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea"
Sep 17 14:07:49.735808 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:49.735790    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea\": container with ID starting with 236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea not found: ID does not exist" containerID="236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea"
Sep 17 14:07:49.735854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.735812    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea"} err="failed to get container status \"236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea\": rpc error: code = NotFound desc = could not find container \"236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea\": container with ID starting with 236212dff1f3fe4dd07aabef3e24bfd9f9523703daa6ebdab9eeea41125f61ea not found: ID does not exist"
Sep 17 14:07:49.735854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.735827    2560 scope.go:117] "RemoveContainer" containerID="6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7"
Sep 17 14:07:49.736038 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:49.736022    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7\": container with ID starting with 6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7 not found: ID does not exist" containerID="6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7"
Sep 17 14:07:49.736091 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.736044    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7"} err="failed to get container status \"6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7\": rpc error: code = NotFound desc = could not find container \"6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7\": container with ID starting with 6ab80d876489e56295d9ecda044e3caea2f351951b9130252993554863b1c1f7 not found: ID does not exist"
Sep 17 14:07:49.736091 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.736065    2560 scope.go:117] "RemoveContainer" containerID="e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807"
Sep 17 14:07:49.736322 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:49.736308    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807\": container with ID starting with e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807 not found: ID does not exist" containerID="e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807"
Sep 17 14:07:49.736420 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.736324    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807"} err="failed to get container status \"e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807\": rpc error: code = NotFound desc = could not find container \"e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807\": container with ID starting with e0868383a4689cf71a58cc413d836c3559c4dae24555405a91eb1920945a8807 not found: ID does not exist"
Sep 17 14:07:49.736420 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.736338    2560 scope.go:117] "RemoveContainer" containerID="70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac"
Sep 17 14:07:49.736583 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:49.736566    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac\": container with ID starting with 70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac not found: ID does not exist" containerID="70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac"
Sep 17 14:07:49.736641 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.736590    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac"} err="failed to get container status \"70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac\": rpc error: code = NotFound desc = could not find container \"70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac\": container with ID starting with 70923e6196f8eac021f8687f211ad625797f595c3891cac933ae4c564044eaac not found: ID does not exist"
Sep 17 14:07:49.736641 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.736609    2560 scope.go:117] "RemoveContainer" containerID="8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea"
Sep 17 14:07:49.736844 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:07:49.736828    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea\": container with ID starting with 8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea not found: ID does not exist" containerID="8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea"
Sep 17 14:07:49.736886 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.736846    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea"} err="failed to get container status \"8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea\": rpc error: code = NotFound desc = could not find container \"8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea\": container with ID starting with 8ce51c4775c2c4228aa15918a6625fa960e45bdd142598f769d93951b9d17fea not found: ID does not exist"
Sep 17 14:07:49.755480 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755447    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:07:49.755728 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755716    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="thanos-sidecar"
Sep 17 14:07:49.755769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755729    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="thanos-sidecar"
Sep 17 14:07:49.755769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755744    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="prometheus"
Sep 17 14:07:49.755769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755750    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="prometheus"
Sep 17 14:07:49.755769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755758    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy-web"
Sep 17 14:07:49.755769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755764    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy-web"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755773    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="config-reloader"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755779    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="config-reloader"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755787    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="init-config-reloader"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755791    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="init-config-reloader"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755803    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy-thanos"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755808    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy-thanos"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755815    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755822    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755861    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="config-reloader"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755870    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="prometheus"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755875    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="thanos-sidecar"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755880    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy-web"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755887    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy-thanos"
Sep 17 14:07:49.755907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.755892    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49c1100-de42-47b0-8370-a41c69b98353" containerName="kube-rbac-proxy"
Sep 17 14:07:49.759244 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.759223    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.764755 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.764731    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jt6qk\""
Sep 17 14:07:49.764890 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.764731    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 14:07:49.764890 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.764839    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 14:07:49.765097 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.765082    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 14:07:49.765375 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.765343    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 14:07:49.767507 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.767489    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 14:07:49.767957 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.767937    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 14:07:49.768013 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.767958    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 14:07:49.768711 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.768696    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 14:07:49.768969 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.768956    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c7qa9d27b95cr\""
Sep 17 14:07:49.769381 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.769350    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 14:07:49.769882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.769868    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 14:07:49.770226 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.770215    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 14:07:49.770732 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.770711    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 14:07:49.771450 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.771437    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 14:07:49.775096 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.775080    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 14:07:49.775825 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.775811    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 14:07:49.786809 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.786777    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:07:49.825440 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825440 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-web-config\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-config\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825591    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-config-out\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825914 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825914 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825914 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825914 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825914 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825914 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825914 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825849    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825914 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.825914 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.826233 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.826233 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.825947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9gt6\" (UniqueName: \"kubernetes.io/projected/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-kube-api-access-x9gt6\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.926862 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.926808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.926862 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.926866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.926891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.926913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.926935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.926957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927017    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927535 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9gt6\" (UniqueName: \"kubernetes.io/projected/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-kube-api-access-x9gt6\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927535 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927173    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927535 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927197    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927535 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927253    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927535 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-web-config\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927535 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-config\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927535 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927535 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-config-out\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927535 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.927535 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.928021 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.928021 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.927964    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.928143 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.928121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.928798 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.928769    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.928915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.928788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.929507 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.929482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.930797 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.930138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.930797 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.930211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.930797 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.930331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.930797 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.930336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.930797 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.930468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.930797 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.930785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.931398 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.931329    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-web-config\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.931538 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.931517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.932058 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.932037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.932700 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.932675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-config-out\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.932787 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.932684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.932983 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.932964    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-config\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.933041 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.932995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:49.940579 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:49.940551    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9gt6\" (UniqueName: \"kubernetes.io/projected/1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f-kube-api-access-x9gt6\") pod \"prometheus-k8s-0\" (UID: \"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:50.068242 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:50.068189    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:07:50.358929 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:50.358847    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:07:50.361319 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:07:50.361290    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5fe8e4_e47b_4858_97b5_cee64e7fbc7f.slice/crio-7096f7aa228fed6f478f1ea7f761f6d5a605a48aebf32a1cc5ae397bb84baa96 WatchSource:0}: Error finding container 7096f7aa228fed6f478f1ea7f761f6d5a605a48aebf32a1cc5ae397bb84baa96: Status 404 returned error can't find the container with id 7096f7aa228fed6f478f1ea7f761f6d5a605a48aebf32a1cc5ae397bb84baa96
Sep 17 14:07:50.687237 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:50.687199    2560 generic.go:356] "Generic (PLEG): container finished" podID="1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f" containerID="e0990b7b8c9319f5536b127b14cdfbc0eb814910801483e97db4b3f76cfae0ac" exitCode=0
Sep 17 14:07:50.687615 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:50.687234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f","Type":"ContainerDied","Data":"e0990b7b8c9319f5536b127b14cdfbc0eb814910801483e97db4b3f76cfae0ac"}
Sep 17 14:07:50.687615 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:50.687307    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f","Type":"ContainerStarted","Data":"7096f7aa228fed6f478f1ea7f761f6d5a605a48aebf32a1cc5ae397bb84baa96"}
Sep 17 14:07:50.972695 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:50.972642    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e49c1100-de42-47b0-8370-a41c69b98353" path="/var/lib/kubelet/pods/e49c1100-de42-47b0-8370-a41c69b98353/volumes"
Sep 17 14:07:51.695256 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:51.695210    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f","Type":"ContainerStarted","Data":"e2af9434d44db63ef9aa9c1af6fcd9941a8aec306c3a27f414193ffb6db75eca"}
Sep 17 14:07:51.695700 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:51.695299    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f","Type":"ContainerStarted","Data":"3f1537553c5a093d766ec33645e0d093d52e2fa8333e6d4cd89c98e057b9f1ec"}
Sep 17 14:07:51.695700 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:51.695339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f","Type":"ContainerStarted","Data":"ad29383018aa6635480ff1b88d170c61ad84e1d0dd2aec80ad7be25a255458f4"}
Sep 17 14:07:51.695700 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:51.695408    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f","Type":"ContainerStarted","Data":"3b8abbf0aaf9eaf9838ff60ca9ed5f7ebc2168d9fba5ad6f8bc792f7a01585bc"}
Sep 17 14:07:51.695700 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:51.695445    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f","Type":"ContainerStarted","Data":"d8c8f608ef67f7e67e0f3a35aaf23733e98ddfd3b2c5703660d70a13ca30231a"}
Sep 17 14:07:51.695700 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:51.695482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f","Type":"ContainerStarted","Data":"8b647ff1c6effde96ca86209a9fda7b42eaa1b1f4b6bfcce72903950e9402873"}
Sep 17 14:07:51.750015 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:51.749955    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.728861086 podStartE2EDuration="2.7499386s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.077514ms" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:51.744668888 +0000 UTC m=+169.335844674" watchObservedRunningTime="2026-09-17 14:07:51.7499386 +0000 UTC m=+169.341114389"
Sep 17 14:07:55.069389 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:07:55.069274    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:08:50.069295 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:08:50.069252    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:08:50.084878 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:08:50.084849    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:08:50.933717 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:08:50.933684    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:02.845949 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:10:02.845914    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 14:11:54.593845 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.593806    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"]
Sep 17 14:11:54.597068 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.597047    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.599615 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.599593    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:11:54.599766 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.599597    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-58dlt\""
Sep 17 14:11:54.600560 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.600545    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 14:11:54.600853 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.600841    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 14:11:54.600908 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.600841    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 14:11:54.600953 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.600907    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 14:11:54.618566 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.618536    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"]
Sep 17 14:11:54.626445 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.626411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c8ff066-59f3-4894-8707-190e212c8c01-cert\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.626600 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.626549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c8ff066-59f3-4894-8707-190e212c8c01-metrics-cert\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.626641 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.626602    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9c8ff066-59f3-4894-8707-190e212c8c01-manager-config\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.626641 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.626626    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqzrh\" (UniqueName: \"kubernetes.io/projected/9c8ff066-59f3-4894-8707-190e212c8c01-kube-api-access-pqzrh\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.727848 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.727809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c8ff066-59f3-4894-8707-190e212c8c01-cert\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.728023 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.727877    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c8ff066-59f3-4894-8707-190e212c8c01-metrics-cert\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.728023 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.727905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9c8ff066-59f3-4894-8707-190e212c8c01-manager-config\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.728023 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.727920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqzrh\" (UniqueName: \"kubernetes.io/projected/9c8ff066-59f3-4894-8707-190e212c8c01-kube-api-access-pqzrh\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.728678 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.728658    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9c8ff066-59f3-4894-8707-190e212c8c01-manager-config\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.730422 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.730393    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c8ff066-59f3-4894-8707-190e212c8c01-metrics-cert\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.730524 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.730424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c8ff066-59f3-4894-8707-190e212c8c01-cert\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.737384 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.737340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqzrh\" (UniqueName: \"kubernetes.io/projected/9c8ff066-59f3-4894-8707-190e212c8c01-kube-api-access-pqzrh\") pod \"lws-controller-manager-5cb7bb5755-54rzn\" (UID: \"9c8ff066-59f3-4894-8707-190e212c8c01\") " pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:54.906566 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:54.906469    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:55.189577 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:55.189498    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"]
Sep 17 14:11:55.192488 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:11:55.192459    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8ff066_59f3_4894_8707_190e212c8c01.slice/crio-16a967603d881008b2270c43dfda6d9cb68922c1d0eff39bfb59f1abc9f1186f WatchSource:0}: Error finding container 16a967603d881008b2270c43dfda6d9cb68922c1d0eff39bfb59f1abc9f1186f: Status 404 returned error can't find the container with id 16a967603d881008b2270c43dfda6d9cb68922c1d0eff39bfb59f1abc9f1186f
Sep 17 14:11:55.194335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:55.194319    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:11:55.603495 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:55.603457    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn" event={"ID":"9c8ff066-59f3-4894-8707-190e212c8c01","Type":"ContainerStarted","Data":"16a967603d881008b2270c43dfda6d9cb68922c1d0eff39bfb59f1abc9f1186f"}
Sep 17 14:11:57.613412 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:57.613348    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn" event={"ID":"9c8ff066-59f3-4894-8707-190e212c8c01","Type":"ContainerStarted","Data":"ce3b01bc0331ef14a84e702a22db9c2bc0720582aa5fee5bdf57499ff37f7671"}
Sep 17 14:11:57.613834 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:57.613510    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:11:57.637841 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:11:57.637795    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn" podStartSLOduration=1.347436013 podStartE2EDuration="3.637776207s" totalImagesPullingTime="2.290340194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:11:57.636838204 +0000 UTC m=+415.228014023" watchObservedRunningTime="2026-09-17 14:11:57.637776207 +0000 UTC m=+415.228951968"
Sep 17 14:12:02.417809 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.417773    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"]
Sep 17 14:12:02.421214 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.421191    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.423872 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.423847    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 14:12:02.423872 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.423863    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 14:12:02.424170 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.424152    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 14:12:02.424254 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.424236    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-89xvx\""
Sep 17 14:12:02.424483 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.424465    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 14:12:02.439445 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.439415    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"]
Sep 17 14:12:02.494923 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.494887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9adba57-c979-487b-81d7-62107b241419-tmp\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.494923 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.494938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9adba57-c979-487b-81d7-62107b241419-webhook-cert\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.495142 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.494972    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xxv\" (UniqueName: \"kubernetes.io/projected/d9adba57-c979-487b-81d7-62107b241419-kube-api-access-92xxv\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.495142 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.494991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9adba57-c979-487b-81d7-62107b241419-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.495142 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.495036    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9adba57-c979-487b-81d7-62107b241419-manifests\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.595771 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.595735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9adba57-c979-487b-81d7-62107b241419-tmp\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.595971 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.595800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9adba57-c979-487b-81d7-62107b241419-webhook-cert\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.595971 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.595847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xxv\" (UniqueName: \"kubernetes.io/projected/d9adba57-c979-487b-81d7-62107b241419-kube-api-access-92xxv\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.595971 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.595875    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9adba57-c979-487b-81d7-62107b241419-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.595971 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.595927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9adba57-c979-487b-81d7-62107b241419-manifests\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.596948 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.596917    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9adba57-c979-487b-81d7-62107b241419-manifests\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.598374 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.598333    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9adba57-c979-487b-81d7-62107b241419-tmp\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.598879 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.598846    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9adba57-c979-487b-81d7-62107b241419-webhook-cert\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.598995 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.598918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9adba57-c979-487b-81d7-62107b241419-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.612503 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.612473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xxv\" (UniqueName: \"kubernetes.io/projected/d9adba57-c979-487b-81d7-62107b241419-kube-api-access-92xxv\") pod \"opendatahub-operator-controller-manager-7895599765-pn78d\" (UID: \"d9adba57-c979-487b-81d7-62107b241419\") " pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:02.731519 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:02.731481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:03.014817 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:03.014791    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"]
Sep 17 14:12:03.017594 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:12:03.017567    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9adba57_c979_487b_81d7_62107b241419.slice/crio-93184d33f11d92f4d5a4febf5371cf76dd4ae67995906971088ffe9c1c8a6e46 WatchSource:0}: Error finding container 93184d33f11d92f4d5a4febf5371cf76dd4ae67995906971088ffe9c1c8a6e46: Status 404 returned error can't find the container with id 93184d33f11d92f4d5a4febf5371cf76dd4ae67995906971088ffe9c1c8a6e46
Sep 17 14:12:03.636966 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:03.636921    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d" event={"ID":"d9adba57-c979-487b-81d7-62107b241419","Type":"ContainerStarted","Data":"93184d33f11d92f4d5a4febf5371cf76dd4ae67995906971088ffe9c1c8a6e46"}
Sep 17 14:12:06.652500 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:06.652417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d" event={"ID":"d9adba57-c979-487b-81d7-62107b241419","Type":"ContainerStarted","Data":"a1575377bf87d7b0ccc4823416f3f8a801726b9db0c4aeab564fb22618205264"}
Sep 17 14:12:07.658704 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:07.658669    2560 generic.go:356] "Generic (PLEG): container finished" podID="d9adba57-c979-487b-81d7-62107b241419" containerID="a1575377bf87d7b0ccc4823416f3f8a801726b9db0c4aeab564fb22618205264" exitCode=0
Sep 17 14:12:07.659090 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:07.658762    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d" event={"ID":"d9adba57-c979-487b-81d7-62107b241419","Type":"ContainerDied","Data":"a1575377bf87d7b0ccc4823416f3f8a801726b9db0c4aeab564fb22618205264"}
Sep 17 14:12:08.619068 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:08.619032    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5cb7bb5755-54rzn"
Sep 17 14:12:08.665153 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:08.665112    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d" event={"ID":"d9adba57-c979-487b-81d7-62107b241419","Type":"ContainerStarted","Data":"8b32e0ddbf253be60fb42e173281d3671a6061f2ad417f335b83354702a3f8d0"}
Sep 17 14:12:08.665544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:08.665278    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:08.692749 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:08.692691    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d" podStartSLOduration=2.020459325 podStartE2EDuration="6.692672218s" totalImagesPullingTime="3.591366164s" totalInitContainerRuntime="1.080846729s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:12:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:12:08.691980071 +0000 UTC m=+426.283155834" watchObservedRunningTime="2026-09-17 14:12:08.692672218 +0000 UTC m=+426.283847979"
Sep 17 14:12:19.671762 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:19.671729    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7895599765-pn78d"
Sep 17 14:12:21.767942 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.767904    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"]
Sep 17 14:12:21.889036 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.888994    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"]
Sep 17 14:12:21.889180 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.889119    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:21.892125 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.892098    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 14:12:21.892125 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.892125    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 14:12:21.892339 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.892152    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 14:12:21.892339 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.892125    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 14:12:21.893027 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.893008    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bt6mz\""
Sep 17 14:12:21.964726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.964690    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/483f6869-a5b2-44e7-9cbc-77ccb3592e3a-tmp\") pod \"kube-auth-proxy-7c6769664b-6wdck\" (UID: \"483f6869-a5b2-44e7-9cbc-77ccb3592e3a\") " pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:21.964901 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.964746    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfb5\" (UniqueName: \"kubernetes.io/projected/483f6869-a5b2-44e7-9cbc-77ccb3592e3a-kube-api-access-4sfb5\") pod \"kube-auth-proxy-7c6769664b-6wdck\" (UID: \"483f6869-a5b2-44e7-9cbc-77ccb3592e3a\") " pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:21.964901 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:21.964805    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/483f6869-a5b2-44e7-9cbc-77ccb3592e3a-tls-certs\") pod \"kube-auth-proxy-7c6769664b-6wdck\" (UID: \"483f6869-a5b2-44e7-9cbc-77ccb3592e3a\") " pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:22.066221 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:22.066133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/483f6869-a5b2-44e7-9cbc-77ccb3592e3a-tmp\") pod \"kube-auth-proxy-7c6769664b-6wdck\" (UID: \"483f6869-a5b2-44e7-9cbc-77ccb3592e3a\") " pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:22.066221 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:22.066187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfb5\" (UniqueName: \"kubernetes.io/projected/483f6869-a5b2-44e7-9cbc-77ccb3592e3a-kube-api-access-4sfb5\") pod \"kube-auth-proxy-7c6769664b-6wdck\" (UID: \"483f6869-a5b2-44e7-9cbc-77ccb3592e3a\") " pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:22.066221 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:22.066217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/483f6869-a5b2-44e7-9cbc-77ccb3592e3a-tls-certs\") pod \"kube-auth-proxy-7c6769664b-6wdck\" (UID: \"483f6869-a5b2-44e7-9cbc-77ccb3592e3a\") " pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:22.068641 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:22.068612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/483f6869-a5b2-44e7-9cbc-77ccb3592e3a-tmp\") pod \"kube-auth-proxy-7c6769664b-6wdck\" (UID: \"483f6869-a5b2-44e7-9cbc-77ccb3592e3a\") " pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:22.068799 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:22.068782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/483f6869-a5b2-44e7-9cbc-77ccb3592e3a-tls-certs\") pod \"kube-auth-proxy-7c6769664b-6wdck\" (UID: \"483f6869-a5b2-44e7-9cbc-77ccb3592e3a\") " pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:22.079881 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:22.079854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfb5\" (UniqueName: \"kubernetes.io/projected/483f6869-a5b2-44e7-9cbc-77ccb3592e3a-kube-api-access-4sfb5\") pod \"kube-auth-proxy-7c6769664b-6wdck\" (UID: \"483f6869-a5b2-44e7-9cbc-77ccb3592e3a\") " pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:22.199020 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:22.198981    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"
Sep 17 14:12:22.489867 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:22.489835    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6769664b-6wdck"]
Sep 17 14:12:22.493783 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:12:22.493755    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483f6869_a5b2_44e7_9cbc_77ccb3592e3a.slice/crio-7f2e727d3cab37e6dda64751fefab8c828e7b768a01d7ab3b83ed8a32ce3badb WatchSource:0}: Error finding container 7f2e727d3cab37e6dda64751fefab8c828e7b768a01d7ab3b83ed8a32ce3badb: Status 404 returned error can't find the container with id 7f2e727d3cab37e6dda64751fefab8c828e7b768a01d7ab3b83ed8a32ce3badb
Sep 17 14:12:22.722018 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:22.721981    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck" event={"ID":"483f6869-a5b2-44e7-9cbc-77ccb3592e3a","Type":"ContainerStarted","Data":"7f2e727d3cab37e6dda64751fefab8c828e7b768a01d7ab3b83ed8a32ce3badb"}
Sep 17 14:12:25.735087 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:25.735047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck" event={"ID":"483f6869-a5b2-44e7-9cbc-77ccb3592e3a","Type":"ContainerStarted","Data":"90201716fcce66fcbda0da89c2b4e2389da2f7941138cb3a878070d8e2515671"}
Sep 17 14:12:25.757107 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:12:25.757049    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c6769664b-6wdck" podStartSLOduration=1.9586755089999999 podStartE2EDuration="4.757032759s" totalImagesPullingTime="2.79835725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:12:25.756456288 +0000 UTC m=+443.347632052" watchObservedRunningTime="2026-09-17 14:12:25.757032759 +0000 UTC m=+443.348208520"
Sep 17 14:14:06.651858 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.651779    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"]
Sep 17 14:14:06.654825 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.654807    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:06.657466 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.657440    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 14:14:06.658583 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.658545    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 14:14:06.658699 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.658591    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 14:14:06.658699 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.658593    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h77dj\""
Sep 17 14:14:06.658699 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.658626    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 14:14:06.664180 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.664157    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"]
Sep 17 14:14:06.769013 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.768970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4804acc-0724-42db-8263-52f1611d5892-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rgqch\" (UID: \"a4804acc-0724-42db-8263-52f1611d5892\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:06.769013 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.769014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4804acc-0724-42db-8263-52f1611d5892-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rgqch\" (UID: \"a4804acc-0724-42db-8263-52f1611d5892\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:06.769224 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.769037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ch7\" (UniqueName: \"kubernetes.io/projected/a4804acc-0724-42db-8263-52f1611d5892-kube-api-access-l9ch7\") pod \"kuadrant-console-plugin-fbdd4f445-rgqch\" (UID: \"a4804acc-0724-42db-8263-52f1611d5892\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:06.870028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.869997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4804acc-0724-42db-8263-52f1611d5892-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rgqch\" (UID: \"a4804acc-0724-42db-8263-52f1611d5892\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:06.870028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.870033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4804acc-0724-42db-8263-52f1611d5892-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rgqch\" (UID: \"a4804acc-0724-42db-8263-52f1611d5892\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:06.870140 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.870056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ch7\" (UniqueName: \"kubernetes.io/projected/a4804acc-0724-42db-8263-52f1611d5892-kube-api-access-l9ch7\") pod \"kuadrant-console-plugin-fbdd4f445-rgqch\" (UID: \"a4804acc-0724-42db-8263-52f1611d5892\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:06.870177 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:14:06.870145    2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 17 14:14:06.870228 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:14:06.870217    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4804acc-0724-42db-8263-52f1611d5892-plugin-serving-cert podName:a4804acc-0724-42db-8263-52f1611d5892 nodeName:}" failed. No retries permitted until 2026-09-17 14:14:07.370201226 +0000 UTC m=+544.961376967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a4804acc-0724-42db-8263-52f1611d5892-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-rgqch" (UID: "a4804acc-0724-42db-8263-52f1611d5892") : secret "plugin-serving-cert" not found
Sep 17 14:14:06.870730 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.870705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4804acc-0724-42db-8263-52f1611d5892-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rgqch\" (UID: \"a4804acc-0724-42db-8263-52f1611d5892\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:06.881211 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:06.881168    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ch7\" (UniqueName: \"kubernetes.io/projected/a4804acc-0724-42db-8263-52f1611d5892-kube-api-access-l9ch7\") pod \"kuadrant-console-plugin-fbdd4f445-rgqch\" (UID: \"a4804acc-0724-42db-8263-52f1611d5892\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:07.374841 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:07.374803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4804acc-0724-42db-8263-52f1611d5892-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rgqch\" (UID: \"a4804acc-0724-42db-8263-52f1611d5892\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:07.380690 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:07.378557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4804acc-0724-42db-8263-52f1611d5892-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rgqch\" (UID: \"a4804acc-0724-42db-8263-52f1611d5892\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:07.564020 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:07.563978    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"
Sep 17 14:14:07.837213 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:07.837186    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch"]
Sep 17 14:14:07.839912 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:14:07.839884    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4804acc_0724_42db_8263_52f1611d5892.slice/crio-70de908a914f86536ab91ba6221740426cbfa9156fbf3317ea5afb76eb7736bc WatchSource:0}: Error finding container 70de908a914f86536ab91ba6221740426cbfa9156fbf3317ea5afb76eb7736bc: Status 404 returned error can't find the container with id 70de908a914f86536ab91ba6221740426cbfa9156fbf3317ea5afb76eb7736bc
Sep 17 14:14:08.137339 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:08.137252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch" event={"ID":"a4804acc-0724-42db-8263-52f1611d5892","Type":"ContainerStarted","Data":"70de908a914f86536ab91ba6221740426cbfa9156fbf3317ea5afb76eb7736bc"}
Sep 17 14:14:13.166337 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:13.166297    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch" event={"ID":"a4804acc-0724-42db-8263-52f1611d5892","Type":"ContainerStarted","Data":"780d2c8ab0e813a6d83287781c0c3c7a60d59d3508743520c097512670e349c4"}
Sep 17 14:14:13.183077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:13.183021    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rgqch" podStartSLOduration=2.727523797 podStartE2EDuration="7.183003819s" totalImagesPullingTime="4.455480022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:14:13.182310807 +0000 UTC m=+550.773486569" watchObservedRunningTime="2026-09-17 14:14:13.183003819 +0000 UTC m=+550.774179580"
Sep 17 14:14:48.516560 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.516523    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pvkzd"]
Sep 17 14:14:48.532910 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.532869    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:14:48.534903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.534859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pvkzd"]
Sep 17 14:14:48.535815 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.535792    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 14:14:48.593413 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.593378    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pvkzd"]
Sep 17 14:14:48.633966 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.633925    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-config-file\") pod \"limitador-limitador-8588749977-pvkzd\" (UID: \"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab\") " pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:14:48.634140 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.633990    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk5ht\" (UniqueName: \"kubernetes.io/projected/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-kube-api-access-wk5ht\") pod \"limitador-limitador-8588749977-pvkzd\" (UID: \"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab\") " pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:14:48.734776 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.734740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-config-file\") pod \"limitador-limitador-8588749977-pvkzd\" (UID: \"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab\") " pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:14:48.734907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.734796    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk5ht\" (UniqueName: \"kubernetes.io/projected/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-kube-api-access-wk5ht\") pod \"limitador-limitador-8588749977-pvkzd\" (UID: \"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab\") " pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:14:48.735410 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.735390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-config-file\") pod \"limitador-limitador-8588749977-pvkzd\" (UID: \"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab\") " pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:14:48.744078 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.744043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk5ht\" (UniqueName: \"kubernetes.io/projected/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-kube-api-access-wk5ht\") pod \"limitador-limitador-8588749977-pvkzd\" (UID: \"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab\") " pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:14:48.845235 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:48.845147    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:14:49.124885 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:49.124804    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pvkzd"]
Sep 17 14:14:49.128369 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:14:49.128328    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81afd07d_2ca6_4ef9_a12d_274c8e1e25ab.slice/crio-1c3b3ff63074986deefba807a7b7b07fece6e0b555b40636185124b29633d097 WatchSource:0}: Error finding container 1c3b3ff63074986deefba807a7b7b07fece6e0b555b40636185124b29633d097: Status 404 returned error can't find the container with id 1c3b3ff63074986deefba807a7b7b07fece6e0b555b40636185124b29633d097
Sep 17 14:14:49.307991 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:49.307953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd" event={"ID":"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab","Type":"ContainerStarted","Data":"1c3b3ff63074986deefba807a7b7b07fece6e0b555b40636185124b29633d097"}
Sep 17 14:14:49.465412 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:49.465319    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2bxjh"]
Sep 17 14:14:49.468208 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:49.468190    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2bxjh"
Sep 17 14:14:49.471254 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:49.471233    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mfw9h\""
Sep 17 14:14:49.484268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:49.484238    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2bxjh"]
Sep 17 14:14:49.641437 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:49.641388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-228ql\" (UniqueName: \"kubernetes.io/projected/98ea51cd-3ef9-4a18-9a40-cd3bef600058-kube-api-access-228ql\") pod \"authorino-59c974bd48-2bxjh\" (UID: \"98ea51cd-3ef9-4a18-9a40-cd3bef600058\") " pod="kuadrant-system/authorino-59c974bd48-2bxjh"
Sep 17 14:14:49.742205 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:49.742162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-228ql\" (UniqueName: \"kubernetes.io/projected/98ea51cd-3ef9-4a18-9a40-cd3bef600058-kube-api-access-228ql\") pod \"authorino-59c974bd48-2bxjh\" (UID: \"98ea51cd-3ef9-4a18-9a40-cd3bef600058\") " pod="kuadrant-system/authorino-59c974bd48-2bxjh"
Sep 17 14:14:49.752618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:49.752583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-228ql\" (UniqueName: \"kubernetes.io/projected/98ea51cd-3ef9-4a18-9a40-cd3bef600058-kube-api-access-228ql\") pod \"authorino-59c974bd48-2bxjh\" (UID: \"98ea51cd-3ef9-4a18-9a40-cd3bef600058\") " pod="kuadrant-system/authorino-59c974bd48-2bxjh"
Sep 17 14:14:49.777706 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:49.777673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2bxjh"
Sep 17 14:14:50.061233 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:50.061113    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2bxjh"]
Sep 17 14:14:50.064498 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:14:50.064468    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98ea51cd_3ef9_4a18_9a40_cd3bef600058.slice/crio-82d87a1e9987b921fb125cc983ed94ae18331e07c017f71302f26d045fdfcab9 WatchSource:0}: Error finding container 82d87a1e9987b921fb125cc983ed94ae18331e07c017f71302f26d045fdfcab9: Status 404 returned error can't find the container with id 82d87a1e9987b921fb125cc983ed94ae18331e07c017f71302f26d045fdfcab9
Sep 17 14:14:50.314969 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:50.314886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2bxjh" event={"ID":"98ea51cd-3ef9-4a18-9a40-cd3bef600058","Type":"ContainerStarted","Data":"82d87a1e9987b921fb125cc983ed94ae18331e07c017f71302f26d045fdfcab9"}
Sep 17 14:14:52.326444 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:52.326398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd" event={"ID":"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab","Type":"ContainerStarted","Data":"b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60"}
Sep 17 14:14:52.326877 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:52.326630    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:14:52.327904 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:52.327886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2bxjh" event={"ID":"98ea51cd-3ef9-4a18-9a40-cd3bef600058","Type":"ContainerStarted","Data":"b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d"}
Sep 17 14:14:52.354083 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:52.354031    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd" podStartSLOduration=2.193300607 podStartE2EDuration="4.35401156s" totalImagesPullingTime="2.160710953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:14:52.353830636 +0000 UTC m=+589.945006397" watchObservedRunningTime="2026-09-17 14:14:52.35401156 +0000 UTC m=+589.945187321"
Sep 17 14:14:52.385431 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:14:52.385372    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2bxjh" podStartSLOduration=1.219695114 podStartE2EDuration="3.385345868s" totalImagesPullingTime="2.165650754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:14:52.383794483 +0000 UTC m=+589.974970246" watchObservedRunningTime="2026-09-17 14:14:52.385345868 +0000 UTC m=+589.976521629"
Sep 17 14:15:02.476235 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:02.476193    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pvkzd"]
Sep 17 14:15:02.476732 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:02.476521    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd" podUID="81afd07d-2ca6-4ef9-a12d-274c8e1e25ab" containerName="limitador" containerID="cri-o://b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60" gracePeriod=30
Sep 17 14:15:02.477209 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:02.477166    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:15:03.022441 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.022414    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:15:03.049334 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.049255    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk5ht\" (UniqueName: \"kubernetes.io/projected/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-kube-api-access-wk5ht\") pod \"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab\" (UID: \"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab\") "
Sep 17 14:15:03.051453 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.051422    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-kube-api-access-wk5ht" (OuterVolumeSpecName: "kube-api-access-wk5ht") pod "81afd07d-2ca6-4ef9-a12d-274c8e1e25ab" (UID: "81afd07d-2ca6-4ef9-a12d-274c8e1e25ab"). InnerVolumeSpecName "kube-api-access-wk5ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:15:03.150039 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.150003    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-config-file\") pod \"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab\" (UID: \"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab\") "
Sep 17 14:15:03.150268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.150248    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk5ht\" (UniqueName: \"kubernetes.io/projected/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-kube-api-access-wk5ht\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:15:03.150345 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.150325    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-config-file" (OuterVolumeSpecName: "config-file") pod "81afd07d-2ca6-4ef9-a12d-274c8e1e25ab" (UID: "81afd07d-2ca6-4ef9-a12d-274c8e1e25ab"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:15:03.251348 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.251314    2560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab-config-file\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:15:03.374271 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.374185    2560 generic.go:356] "Generic (PLEG): container finished" podID="81afd07d-2ca6-4ef9-a12d-274c8e1e25ab" containerID="b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60" exitCode=0
Sep 17 14:15:03.374271 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.374223    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd"
Sep 17 14:15:03.374271 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.374263    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd" event={"ID":"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab","Type":"ContainerDied","Data":"b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60"}
Sep 17 14:15:03.374512 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.374321    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-pvkzd" event={"ID":"81afd07d-2ca6-4ef9-a12d-274c8e1e25ab","Type":"ContainerDied","Data":"1c3b3ff63074986deefba807a7b7b07fece6e0b555b40636185124b29633d097"}
Sep 17 14:15:03.374512 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.374348    2560 scope.go:117] "RemoveContainer" containerID="b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60"
Sep 17 14:15:03.383448 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.383431    2560 scope.go:117] "RemoveContainer" containerID="b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60"
Sep 17 14:15:03.383735 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:15:03.383718    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60\": container with ID starting with b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60 not found: ID does not exist" containerID="b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60"
Sep 17 14:15:03.383775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.383746    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60"} err="failed to get container status \"b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60\": rpc error: code = NotFound desc = could not find container \"b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60\": container with ID starting with b06e48dfd32fb3eaa30127196e86d6678497184ac354efea00230d3899945d60 not found: ID does not exist"
Sep 17 14:15:03.395526 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.395492    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pvkzd"]
Sep 17 14:15:03.399092 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:03.399068    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-pvkzd"]
Sep 17 14:15:04.964886 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:04.964857    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81afd07d-2ca6-4ef9-a12d-274c8e1e25ab" path="/var/lib/kubelet/pods/81afd07d-2ca6-4ef9-a12d-274c8e1e25ab/volumes"
Sep 17 14:15:17.063481 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.063442    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-flz6w"]
Sep 17 14:15:17.063895 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.063755    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81afd07d-2ca6-4ef9-a12d-274c8e1e25ab" containerName="limitador"
Sep 17 14:15:17.063895 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.063768    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="81afd07d-2ca6-4ef9-a12d-274c8e1e25ab" containerName="limitador"
Sep 17 14:15:17.063895 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.063843    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="81afd07d-2ca6-4ef9-a12d-274c8e1e25ab" containerName="limitador"
Sep 17 14:15:17.066515 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.066494    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-flz6w"
Sep 17 14:15:17.083689 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.083648    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-flz6w"]
Sep 17 14:15:17.166467 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.166423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfm79\" (UniqueName: \"kubernetes.io/projected/7e44f059-e932-45db-9ccd-902472ca5edc-kube-api-access-qfm79\") pod \"authorino-557f876f69-flz6w\" (UID: \"7e44f059-e932-45db-9ccd-902472ca5edc\") " pod="kuadrant-system/authorino-557f876f69-flz6w"
Sep 17 14:15:17.267348 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.267305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfm79\" (UniqueName: \"kubernetes.io/projected/7e44f059-e932-45db-9ccd-902472ca5edc-kube-api-access-qfm79\") pod \"authorino-557f876f69-flz6w\" (UID: \"7e44f059-e932-45db-9ccd-902472ca5edc\") " pod="kuadrant-system/authorino-557f876f69-flz6w"
Sep 17 14:15:17.277074 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.277041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfm79\" (UniqueName: \"kubernetes.io/projected/7e44f059-e932-45db-9ccd-902472ca5edc-kube-api-access-qfm79\") pod \"authorino-557f876f69-flz6w\" (UID: \"7e44f059-e932-45db-9ccd-902472ca5edc\") " pod="kuadrant-system/authorino-557f876f69-flz6w"
Sep 17 14:15:17.305705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.305669    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-flz6w"]
Sep 17 14:15:17.305939 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.305927    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-flz6w"
Sep 17 14:15:17.334293 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.334256    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84f6f5fd4-t4qvq"]
Sep 17 14:15:17.337215 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.337189    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84f6f5fd4-t4qvq"
Sep 17 14:15:17.348921 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.348889    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84f6f5fd4-t4qvq"]
Sep 17 14:15:17.368254 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.368218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxck\" (UniqueName: \"kubernetes.io/projected/36a99968-7749-4ea4-b20f-610566eca96f-kube-api-access-5dxck\") pod \"authorino-84f6f5fd4-t4qvq\" (UID: \"36a99968-7749-4ea4-b20f-610566eca96f\") " pod="kuadrant-system/authorino-84f6f5fd4-t4qvq"
Sep 17 14:15:17.407156 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.407115    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84f6f5fd4-t4qvq"]
Sep 17 14:15:17.407509 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:15:17.407484    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5dxck], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-84f6f5fd4-t4qvq" podUID="36a99968-7749-4ea4-b20f-610566eca96f"
Sep 17 14:15:17.435519 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.435489    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84f6f5fd4-t4qvq"
Sep 17 14:15:17.441632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.441604    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84f6f5fd4-t4qvq"
Sep 17 14:15:17.468981 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.468943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxck\" (UniqueName: \"kubernetes.io/projected/36a99968-7749-4ea4-b20f-610566eca96f-kube-api-access-5dxck\") pod \"authorino-84f6f5fd4-t4qvq\" (UID: \"36a99968-7749-4ea4-b20f-610566eca96f\") " pod="kuadrant-system/authorino-84f6f5fd4-t4qvq"
Sep 17 14:15:17.477382 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.477329    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxck\" (UniqueName: \"kubernetes.io/projected/36a99968-7749-4ea4-b20f-610566eca96f-kube-api-access-5dxck\") pod \"authorino-84f6f5fd4-t4qvq\" (UID: \"36a99968-7749-4ea4-b20f-610566eca96f\") " pod="kuadrant-system/authorino-84f6f5fd4-t4qvq"
Sep 17 14:15:17.569825 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.569740    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dxck\" (UniqueName: \"kubernetes.io/projected/36a99968-7749-4ea4-b20f-610566eca96f-kube-api-access-5dxck\") pod \"36a99968-7749-4ea4-b20f-610566eca96f\" (UID: \"36a99968-7749-4ea4-b20f-610566eca96f\") "
Sep 17 14:15:17.571819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.571776    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a99968-7749-4ea4-b20f-610566eca96f-kube-api-access-5dxck" (OuterVolumeSpecName: "kube-api-access-5dxck") pod "36a99968-7749-4ea4-b20f-610566eca96f" (UID: "36a99968-7749-4ea4-b20f-610566eca96f"). InnerVolumeSpecName "kube-api-access-5dxck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:15:17.583190 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.583073    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-flz6w"]
Sep 17 14:15:17.585945 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:15:17.585914    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e44f059_e932_45db_9ccd_902472ca5edc.slice/crio-04fca62216f11747e3f30dad761451d7688ea4c0bb95968b359f233303d42773 WatchSource:0}: Error finding container 04fca62216f11747e3f30dad761451d7688ea4c0bb95968b359f233303d42773: Status 404 returned error can't find the container with id 04fca62216f11747e3f30dad761451d7688ea4c0bb95968b359f233303d42773
Sep 17 14:15:17.670679 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:17.670640    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dxck\" (UniqueName: \"kubernetes.io/projected/36a99968-7749-4ea4-b20f-610566eca96f-kube-api-access-5dxck\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:15:18.442031 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.441984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-flz6w" event={"ID":"7e44f059-e932-45db-9ccd-902472ca5edc","Type":"ContainerStarted","Data":"698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5"}
Sep 17 14:15:18.442031 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.442020    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84f6f5fd4-t4qvq"
Sep 17 14:15:18.442031 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.442029    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-flz6w" podUID="7e44f059-e932-45db-9ccd-902472ca5edc" containerName="authorino" containerID="cri-o://698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5" gracePeriod=30
Sep 17 14:15:18.442577 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.442041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-flz6w" event={"ID":"7e44f059-e932-45db-9ccd-902472ca5edc","Type":"ContainerStarted","Data":"04fca62216f11747e3f30dad761451d7688ea4c0bb95968b359f233303d42773"}
Sep 17 14:15:18.459248 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.459185    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-flz6w" podStartSLOduration=0.811039164 podStartE2EDuration="1.459161985s" totalImagesPullingTime="648.122821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:18.458920847 +0000 UTC m=+616.050096610" watchObservedRunningTime="2026-09-17 14:15:18.459161985 +0000 UTC m=+616.050337748"
Sep 17 14:15:18.529715 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.529618    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84f6f5fd4-t4qvq"]
Sep 17 14:15:18.533618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.533583    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-84f6f5fd4-t4qvq"]
Sep 17 14:15:18.686069 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.686042    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-flz6w"
Sep 17 14:15:18.781104 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.781004    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfm79\" (UniqueName: \"kubernetes.io/projected/7e44f059-e932-45db-9ccd-902472ca5edc-kube-api-access-qfm79\") pod \"7e44f059-e932-45db-9ccd-902472ca5edc\" (UID: \"7e44f059-e932-45db-9ccd-902472ca5edc\") "
Sep 17 14:15:18.783305 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.783270    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e44f059-e932-45db-9ccd-902472ca5edc-kube-api-access-qfm79" (OuterVolumeSpecName: "kube-api-access-qfm79") pod "7e44f059-e932-45db-9ccd-902472ca5edc" (UID: "7e44f059-e932-45db-9ccd-902472ca5edc"). InnerVolumeSpecName "kube-api-access-qfm79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:15:18.882003 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.881963    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfm79\" (UniqueName: \"kubernetes.io/projected/7e44f059-e932-45db-9ccd-902472ca5edc-kube-api-access-qfm79\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:15:18.964786 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:18.964755    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a99968-7749-4ea4-b20f-610566eca96f" path="/var/lib/kubelet/pods/36a99968-7749-4ea4-b20f-610566eca96f/volumes"
Sep 17 14:15:19.448078 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.448041    2560 generic.go:356] "Generic (PLEG): container finished" podID="7e44f059-e932-45db-9ccd-902472ca5edc" containerID="698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5" exitCode=0
Sep 17 14:15:19.448564 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.448087    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-flz6w"
Sep 17 14:15:19.448564 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.448120    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-flz6w" event={"ID":"7e44f059-e932-45db-9ccd-902472ca5edc","Type":"ContainerDied","Data":"698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5"}
Sep 17 14:15:19.448564 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.448161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-flz6w" event={"ID":"7e44f059-e932-45db-9ccd-902472ca5edc","Type":"ContainerDied","Data":"04fca62216f11747e3f30dad761451d7688ea4c0bb95968b359f233303d42773"}
Sep 17 14:15:19.448564 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.448187    2560 scope.go:117] "RemoveContainer" containerID="698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5"
Sep 17 14:15:19.456619 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.456598    2560 scope.go:117] "RemoveContainer" containerID="698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5"
Sep 17 14:15:19.456968 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:15:19.456948    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5\": container with ID starting with 698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5 not found: ID does not exist" containerID="698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5"
Sep 17 14:15:19.457012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.456981    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5"} err="failed to get container status \"698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5\": rpc error: code = NotFound desc = could not find container \"698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5\": container with ID starting with 698d901d09a59700274d3904eb317428feeae7c2c6ebbb68c77a8492dfe4fdb5 not found: ID does not exist"
Sep 17 14:15:19.465148 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.465114    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-flz6w"]
Sep 17 14:15:19.469312 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.469280    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-flz6w"]
Sep 17 14:15:19.567853 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.567818    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2bxjh"]
Sep 17 14:15:19.568021 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.567996    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2bxjh" podUID="98ea51cd-3ef9-4a18-9a40-cd3bef600058" containerName="authorino" containerID="cri-o://b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d" gracePeriod=30
Sep 17 14:15:19.813325 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.813294    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2bxjh"
Sep 17 14:15:19.890705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.890661    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-228ql\" (UniqueName: \"kubernetes.io/projected/98ea51cd-3ef9-4a18-9a40-cd3bef600058-kube-api-access-228ql\") pod \"98ea51cd-3ef9-4a18-9a40-cd3bef600058\" (UID: \"98ea51cd-3ef9-4a18-9a40-cd3bef600058\") "
Sep 17 14:15:19.892763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.892734    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ea51cd-3ef9-4a18-9a40-cd3bef600058-kube-api-access-228ql" (OuterVolumeSpecName: "kube-api-access-228ql") pod "98ea51cd-3ef9-4a18-9a40-cd3bef600058" (UID: "98ea51cd-3ef9-4a18-9a40-cd3bef600058"). InnerVolumeSpecName "kube-api-access-228ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:15:19.991623 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:19.991581    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-228ql\" (UniqueName: \"kubernetes.io/projected/98ea51cd-3ef9-4a18-9a40-cd3bef600058-kube-api-access-228ql\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:15:20.454010 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.453923    2560 generic.go:356] "Generic (PLEG): container finished" podID="98ea51cd-3ef9-4a18-9a40-cd3bef600058" containerID="b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d" exitCode=0
Sep 17 14:15:20.454010 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.453969    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2bxjh"
Sep 17 14:15:20.454556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.454008    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2bxjh" event={"ID":"98ea51cd-3ef9-4a18-9a40-cd3bef600058","Type":"ContainerDied","Data":"b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d"}
Sep 17 14:15:20.454556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.454043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2bxjh" event={"ID":"98ea51cd-3ef9-4a18-9a40-cd3bef600058","Type":"ContainerDied","Data":"82d87a1e9987b921fb125cc983ed94ae18331e07c017f71302f26d045fdfcab9"}
Sep 17 14:15:20.454556 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.454064    2560 scope.go:117] "RemoveContainer" containerID="b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d"
Sep 17 14:15:20.462960 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.462937    2560 scope.go:117] "RemoveContainer" containerID="b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d"
Sep 17 14:15:20.463305 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:15:20.463283    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d\": container with ID starting with b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d not found: ID does not exist" containerID="b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d"
Sep 17 14:15:20.463384 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.463316    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d"} err="failed to get container status \"b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d\": rpc error: code = NotFound desc = could not find container \"b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d\": container with ID starting with b62e42af23666137a396ac7cfb85a7db147d91a6ed1125837193147623e07e4d not found: ID does not exist"
Sep 17 14:15:20.477461 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.477428    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2bxjh"]
Sep 17 14:15:20.482107 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.482078    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2bxjh"]
Sep 17 14:15:20.965176 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.965139    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e44f059-e932-45db-9ccd-902472ca5edc" path="/var/lib/kubelet/pods/7e44f059-e932-45db-9ccd-902472ca5edc/volumes"
Sep 17 14:15:20.965502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:20.965488    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ea51cd-3ef9-4a18-9a40-cd3bef600058" path="/var/lib/kubelet/pods/98ea51cd-3ef9-4a18-9a40-cd3bef600058/volumes"
Sep 17 14:15:24.423851 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.423808    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-56p2c"]
Sep 17 14:15:24.424312 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.424159    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ea51cd-3ef9-4a18-9a40-cd3bef600058" containerName="authorino"
Sep 17 14:15:24.424312 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.424170    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ea51cd-3ef9-4a18-9a40-cd3bef600058" containerName="authorino"
Sep 17 14:15:24.424312 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.424178    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e44f059-e932-45db-9ccd-902472ca5edc" containerName="authorino"
Sep 17 14:15:24.424312 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.424183    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e44f059-e932-45db-9ccd-902472ca5edc" containerName="authorino"
Sep 17 14:15:24.424312 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.424245    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e44f059-e932-45db-9ccd-902472ca5edc" containerName="authorino"
Sep 17 14:15:24.424312 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.424258    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ea51cd-3ef9-4a18-9a40-cd3bef600058" containerName="authorino"
Sep 17 14:15:24.426990 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.426973    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.429615 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.429591    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 14:15:24.429749 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.429705    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p95mp\""
Sep 17 14:15:24.430905 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.430881    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 14:15:24.431036 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.430915    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 14:15:24.436888 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.436863    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-56p2c"]
Sep 17 14:15:24.530654 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.530568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-metrics-certs\") pod \"maas-controller-5894d6dbc9-56p2c\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") " pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.530654 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.530605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxcm\" (UniqueName: \"kubernetes.io/projected/4e746211-87ab-4c76-9f69-0418846181b6-kube-api-access-kdxcm\") pod \"maas-controller-5894d6dbc9-56p2c\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") " pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.530654 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.530638    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-webhook-cert\") pod \"maas-controller-5894d6dbc9-56p2c\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") " pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.613244 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.613196    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-slhtt"]
Sep 17 14:15:24.615544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.615525    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:24.625541 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.625514    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-slhtt"]
Sep 17 14:15:24.631298 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.631271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-metrics-certs\") pod \"maas-controller-5894d6dbc9-56p2c\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") " pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.631453 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.631302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxcm\" (UniqueName: \"kubernetes.io/projected/4e746211-87ab-4c76-9f69-0418846181b6-kube-api-access-kdxcm\") pod \"maas-controller-5894d6dbc9-56p2c\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") " pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.631453 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.631329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-webhook-cert\") pod \"maas-controller-5894d6dbc9-56p2c\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") " pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.633801 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.633775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-metrics-certs\") pod \"maas-controller-5894d6dbc9-56p2c\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") " pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.633911 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.633816    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-webhook-cert\") pod \"maas-controller-5894d6dbc9-56p2c\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") " pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.639318 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.639293    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxcm\" (UniqueName: \"kubernetes.io/projected/4e746211-87ab-4c76-9f69-0418846181b6-kube-api-access-kdxcm\") pod \"maas-controller-5894d6dbc9-56p2c\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") " pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.731886 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.731859    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-metrics-certs\") pod \"maas-controller-6b6bd7d7cd-slhtt\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") " pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:24.732058 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.731898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-webhook-cert\") pod \"maas-controller-6b6bd7d7cd-slhtt\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") " pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:24.732058 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.731941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6wdc\" (UniqueName: \"kubernetes.io/projected/b391ea6c-16e1-4913-bae6-c4944296a762-kube-api-access-m6wdc\") pod \"maas-controller-6b6bd7d7cd-slhtt\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") " pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:24.737382 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.737334    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:24.833025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.832974    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-metrics-certs\") pod \"maas-controller-6b6bd7d7cd-slhtt\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") " pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:24.833216 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.833034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-webhook-cert\") pod \"maas-controller-6b6bd7d7cd-slhtt\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") " pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:24.833216 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.833109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6wdc\" (UniqueName: \"kubernetes.io/projected/b391ea6c-16e1-4913-bae6-c4944296a762-kube-api-access-m6wdc\") pod \"maas-controller-6b6bd7d7cd-slhtt\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") " pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:24.835743 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.835719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-metrics-certs\") pod \"maas-controller-6b6bd7d7cd-slhtt\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") " pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:24.835834 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.835743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-webhook-cert\") pod \"maas-controller-6b6bd7d7cd-slhtt\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") " pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:24.841981 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.841945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6wdc\" (UniqueName: \"kubernetes.io/projected/b391ea6c-16e1-4913-bae6-c4944296a762-kube-api-access-m6wdc\") pod \"maas-controller-6b6bd7d7cd-slhtt\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") " pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:24.926281 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:24.926237    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:25.014827 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:25.014794    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-56p2c"]
Sep 17 14:15:25.016458 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:15:25.016434    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e746211_87ab_4c76_9f69_0418846181b6.slice/crio-b28e2f8f8a63d82a66e995208f2aab56b7ec952dceaa2af48f13a92b23f84c65 WatchSource:0}: Error finding container b28e2f8f8a63d82a66e995208f2aab56b7ec952dceaa2af48f13a92b23f84c65: Status 404 returned error can't find the container with id b28e2f8f8a63d82a66e995208f2aab56b7ec952dceaa2af48f13a92b23f84c65
Sep 17 14:15:25.208216 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:25.208146    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-slhtt"]
Sep 17 14:15:25.210978 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:15:25.210947    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb391ea6c_16e1_4913_bae6_c4944296a762.slice/crio-aed1fcc5e1b7940f8edc7ed9249b93441c13ac1f5b5bbf0ba77f4f54b5e28a73 WatchSource:0}: Error finding container aed1fcc5e1b7940f8edc7ed9249b93441c13ac1f5b5bbf0ba77f4f54b5e28a73: Status 404 returned error can't find the container with id aed1fcc5e1b7940f8edc7ed9249b93441c13ac1f5b5bbf0ba77f4f54b5e28a73
Sep 17 14:15:25.484530 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:25.484494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5894d6dbc9-56p2c" event={"ID":"4e746211-87ab-4c76-9f69-0418846181b6","Type":"ContainerStarted","Data":"b28e2f8f8a63d82a66e995208f2aab56b7ec952dceaa2af48f13a92b23f84c65"}
Sep 17 14:15:25.485648 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:25.485621    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt" event={"ID":"b391ea6c-16e1-4913-bae6-c4944296a762","Type":"ContainerStarted","Data":"aed1fcc5e1b7940f8edc7ed9249b93441c13ac1f5b5bbf0ba77f4f54b5e28a73"}
Sep 17 14:15:28.504230 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:28.504190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5894d6dbc9-56p2c" event={"ID":"4e746211-87ab-4c76-9f69-0418846181b6","Type":"ContainerStarted","Data":"be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5"}
Sep 17 14:15:28.504678 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:28.504297    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:28.506158 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:28.506126    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt" event={"ID":"b391ea6c-16e1-4913-bae6-c4944296a762","Type":"ContainerStarted","Data":"ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504"}
Sep 17 14:15:28.506316 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:28.506208    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:28.526887 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:28.526829    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5894d6dbc9-56p2c" podStartSLOduration=1.6473111870000001 podStartE2EDuration="4.526811229s" totalImagesPullingTime="2.879500042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:28.525949205 +0000 UTC m=+626.117124967" watchObservedRunningTime="2026-09-17 14:15:28.526811229 +0000 UTC m=+626.117986994"
Sep 17 14:15:28.549365 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:28.549284    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt" podStartSLOduration=1.874094959 podStartE2EDuration="4.549261031s" totalImagesPullingTime="2.675166072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:28.547795285 +0000 UTC m=+626.138971078" watchObservedRunningTime="2026-09-17 14:15:28.549261031 +0000 UTC m=+626.140436794"
Sep 17 14:15:39.512854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.512821    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:15:39.513232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.512890    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:39.599445 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.599410    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-56p2c"]
Sep 17 14:15:39.599648 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.599601    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5894d6dbc9-56p2c" podUID="4e746211-87ab-4c76-9f69-0418846181b6" containerName="manager" containerID="cri-o://be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5" gracePeriod=30
Sep 17 14:15:39.849335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.849305    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:39.972386 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.972316    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdxcm\" (UniqueName: \"kubernetes.io/projected/4e746211-87ab-4c76-9f69-0418846181b6-kube-api-access-kdxcm\") pod \"4e746211-87ab-4c76-9f69-0418846181b6\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") "
Sep 17 14:15:39.972532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.972475    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-webhook-cert\") pod \"4e746211-87ab-4c76-9f69-0418846181b6\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") "
Sep 17 14:15:39.972532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.972508    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-metrics-certs\") pod \"4e746211-87ab-4c76-9f69-0418846181b6\" (UID: \"4e746211-87ab-4c76-9f69-0418846181b6\") "
Sep 17 14:15:39.974633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.974589    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4e746211-87ab-4c76-9f69-0418846181b6" (UID: "4e746211-87ab-4c76-9f69-0418846181b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:15:39.974768 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.974669    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4e746211-87ab-4c76-9f69-0418846181b6" (UID: "4e746211-87ab-4c76-9f69-0418846181b6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:15:39.974768 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:39.974668    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e746211-87ab-4c76-9f69-0418846181b6-kube-api-access-kdxcm" (OuterVolumeSpecName: "kube-api-access-kdxcm") pod "4e746211-87ab-4c76-9f69-0418846181b6" (UID: "4e746211-87ab-4c76-9f69-0418846181b6"). InnerVolumeSpecName "kube-api-access-kdxcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:15:40.073896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.073803    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-webhook-cert\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:15:40.073896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.073835    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e746211-87ab-4c76-9f69-0418846181b6-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:15:40.073896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.073846    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdxcm\" (UniqueName: \"kubernetes.io/projected/4e746211-87ab-4c76-9f69-0418846181b6-kube-api-access-kdxcm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:15:40.560758 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.560718    2560 generic.go:356] "Generic (PLEG): container finished" podID="4e746211-87ab-4c76-9f69-0418846181b6" containerID="be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5" exitCode=0
Sep 17 14:15:40.560758 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.560762    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5894d6dbc9-56p2c"
Sep 17 14:15:40.561262 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.560778    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5894d6dbc9-56p2c" event={"ID":"4e746211-87ab-4c76-9f69-0418846181b6","Type":"ContainerDied","Data":"be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5"}
Sep 17 14:15:40.561262 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.560835    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5894d6dbc9-56p2c" event={"ID":"4e746211-87ab-4c76-9f69-0418846181b6","Type":"ContainerDied","Data":"b28e2f8f8a63d82a66e995208f2aab56b7ec952dceaa2af48f13a92b23f84c65"}
Sep 17 14:15:40.561262 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.560864    2560 scope.go:117] "RemoveContainer" containerID="be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5"
Sep 17 14:15:40.570149 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.570125    2560 scope.go:117] "RemoveContainer" containerID="be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5"
Sep 17 14:15:40.570588 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:15:40.570562    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5\": container with ID starting with be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5 not found: ID does not exist" containerID="be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5"
Sep 17 14:15:40.570656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.570604    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5"} err="failed to get container status \"be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5\": rpc error: code = NotFound desc = could not find container \"be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5\": container with ID starting with be837a066a16c6e3ebef8740ab9ae673478cda4d0602cab62b239b322e726ed5 not found: ID does not exist"
Sep 17 14:15:40.584858 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.584819    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-56p2c"]
Sep 17 14:15:40.597797 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.597761    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-56p2c"]
Sep 17 14:15:40.965861 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:15:40.965775    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e746211-87ab-4c76-9f69-0418846181b6" path="/var/lib/kubelet/pods/4e746211-87ab-4c76-9f69-0418846181b6/volumes"
Sep 17 14:16:01.279332 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.279296    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"]
Sep 17 14:16:01.280765 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.279630    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e746211-87ab-4c76-9f69-0418846181b6" containerName="manager"
Sep 17 14:16:01.280765 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.279641    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e746211-87ab-4c76-9f69-0418846181b6" containerName="manager"
Sep 17 14:16:01.280765 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.279700    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e746211-87ab-4c76-9f69-0418846181b6" containerName="manager"
Sep 17 14:16:01.281444 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.281428    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.284924 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.284899    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 14:16:01.284924 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.284925    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 14:16:01.285145 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.284994    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 17 14:16:01.285210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.285197    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:16:01.285845 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.285831    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d65tq\""
Sep 17 14:16:01.294336 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.294309    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"]
Sep 17 14:16:01.359880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.359848    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-metrics-certs\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.359880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.359884    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b160c3cc-fc7b-464f-9760-5c5d299f62fb-openshift-service-ca\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.360096 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.359946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgx4k\" (UniqueName: \"kubernetes.io/projected/b160c3cc-fc7b-464f-9760-5c5d299f62fb-kube-api-access-sgx4k\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.360096 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.360003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-maas-api-tls\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.461482 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.461445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-metrics-certs\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.461482 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.461491    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b160c3cc-fc7b-464f-9760-5c5d299f62fb-openshift-service-ca\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.461754 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.461539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgx4k\" (UniqueName: \"kubernetes.io/projected/b160c3cc-fc7b-464f-9760-5c5d299f62fb-kube-api-access-sgx4k\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.461754 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.461576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-maas-api-tls\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.461754 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:16:01.461686    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 17 14:16:01.461902 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:16:01.461789    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-maas-api-tls podName:b160c3cc-fc7b-464f-9760-5c5d299f62fb nodeName:}" failed. No retries permitted until 2026-09-17 14:16:01.961766004 +0000 UTC m=+659.552941747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-maas-api-tls") pod "maas-api-5ffb5cf8dc-f2bcg" (UID: "b160c3cc-fc7b-464f-9760-5c5d299f62fb") : secret "maas-api-serving-cert" not found
Sep 17 14:16:01.462220 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.462199    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b160c3cc-fc7b-464f-9760-5c5d299f62fb-openshift-service-ca\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.464156 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.464130    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-metrics-certs\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.473168 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.473143    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgx4k\" (UniqueName: \"kubernetes.io/projected/b160c3cc-fc7b-464f-9760-5c5d299f62fb-kube-api-access-sgx4k\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.894961 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.894925    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"]
Sep 17 14:16:01.901854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.901826    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"
Sep 17 14:16:01.904921 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.904895    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-chv8d\""
Sep 17 14:16:01.905059 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.904895    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 17 14:16:01.912889 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.912853    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"]
Sep 17 14:16:01.966787 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.966740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-maas-api-tls\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:01.966965 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.966832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17506b00-b587-4549-a48c-a11211e2a6bd-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zvgmc\" (UID: \"17506b00-b587-4549-a48c-a11211e2a6bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"
Sep 17 14:16:01.966965 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.966864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl77k\" (UniqueName: \"kubernetes.io/projected/17506b00-b587-4549-a48c-a11211e2a6bd-kube-api-access-xl77k\") pod \"payload-processing-5b54b7fbb7-zvgmc\" (UID: \"17506b00-b587-4549-a48c-a11211e2a6bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"
Sep 17 14:16:01.969263 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:01.969232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-maas-api-tls\") pod \"maas-api-5ffb5cf8dc-f2bcg\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:02.067423 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:02.067342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17506b00-b587-4549-a48c-a11211e2a6bd-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zvgmc\" (UID: \"17506b00-b587-4549-a48c-a11211e2a6bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"
Sep 17 14:16:02.067423 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:02.067427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl77k\" (UniqueName: \"kubernetes.io/projected/17506b00-b587-4549-a48c-a11211e2a6bd-kube-api-access-xl77k\") pod \"payload-processing-5b54b7fbb7-zvgmc\" (UID: \"17506b00-b587-4549-a48c-a11211e2a6bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"
Sep 17 14:16:02.068038 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:02.068019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17506b00-b587-4549-a48c-a11211e2a6bd-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zvgmc\" (UID: \"17506b00-b587-4549-a48c-a11211e2a6bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"
Sep 17 14:16:02.076756 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:02.076725    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl77k\" (UniqueName: \"kubernetes.io/projected/17506b00-b587-4549-a48c-a11211e2a6bd-kube-api-access-xl77k\") pod \"payload-processing-5b54b7fbb7-zvgmc\" (UID: \"17506b00-b587-4549-a48c-a11211e2a6bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"
Sep 17 14:16:02.192384 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:02.192256    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:02.213488 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:02.213441    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"
Sep 17 14:16:02.501173 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:02.501143    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"]
Sep 17 14:16:02.503938 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:16:02.503912    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb160c3cc_fc7b_464f_9760_5c5d299f62fb.slice/crio-6811794ca8af6aaf01023cf94d06c860a3d1330d99383210f496b90563965f4e WatchSource:0}: Error finding container 6811794ca8af6aaf01023cf94d06c860a3d1330d99383210f496b90563965f4e: Status 404 returned error can't find the container with id 6811794ca8af6aaf01023cf94d06c860a3d1330d99383210f496b90563965f4e
Sep 17 14:16:02.531283 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:02.531259    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"]
Sep 17 14:16:02.533757 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:16:02.533724    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17506b00_b587_4549_a48c_a11211e2a6bd.slice/crio-f8337520b79fb6da81e5e855d46da6f92d96852586874a4a3d4f2ab68016f16c WatchSource:0}: Error finding container f8337520b79fb6da81e5e855d46da6f92d96852586874a4a3d4f2ab68016f16c: Status 404 returned error can't find the container with id f8337520b79fb6da81e5e855d46da6f92d96852586874a4a3d4f2ab68016f16c
Sep 17 14:16:02.659502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:02.659465    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg" event={"ID":"b160c3cc-fc7b-464f-9760-5c5d299f62fb","Type":"ContainerStarted","Data":"6811794ca8af6aaf01023cf94d06c860a3d1330d99383210f496b90563965f4e"}
Sep 17 14:16:02.660903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:02.660874    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc" event={"ID":"17506b00-b587-4549-a48c-a11211e2a6bd","Type":"ContainerStarted","Data":"f8337520b79fb6da81e5e855d46da6f92d96852586874a4a3d4f2ab68016f16c"}
Sep 17 14:16:05.681279 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:05.681158    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg" event={"ID":"b160c3cc-fc7b-464f-9760-5c5d299f62fb","Type":"ContainerStarted","Data":"b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5"}
Sep 17 14:16:05.681279 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:05.681235    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:05.682872 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:05.682846    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc" event={"ID":"17506b00-b587-4549-a48c-a11211e2a6bd","Type":"ContainerStarted","Data":"44dd331191bd5bdeda2ba7cfec7ac8e00bf989ca70a8e605a0f98478e47493d4"}
Sep 17 14:16:05.682975 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:05.682907    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"
Sep 17 14:16:05.703351 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:05.703293    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg" podStartSLOduration=1.906765297 podStartE2EDuration="4.703275503s" totalImagesPullingTime="2.796510206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:05.701344979 +0000 UTC m=+663.292520743" watchObservedRunningTime="2026-09-17 14:16:05.703275503 +0000 UTC m=+663.294451265"
Sep 17 14:16:05.721904 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:05.721848    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc" podStartSLOduration=1.908694003 podStartE2EDuration="4.721827684s" totalImagesPullingTime="2.813133681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:05.720329268 +0000 UTC m=+663.311505031" watchObservedRunningTime="2026-09-17 14:16:05.721827684 +0000 UTC m=+663.313003445"
Sep 17 14:16:11.692563 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:11.692533    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:16:16.689198 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:16.689165    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zvgmc"
Sep 17 14:16:18.497901 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.497864    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"]
Sep 17 14:16:18.501677 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.501650    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.504238 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.504211    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 14:16:18.504581 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.504560    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dg4pv\""
Sep 17 14:16:18.505459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.505435    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 14:16:18.505557 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.505435    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 17 14:16:18.511873 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.511844    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"]
Sep 17 14:16:18.617704 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.617666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.617905 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.617713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1228c3-5493-4572-baef-43101267ddaa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.617905 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.617743    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.617905 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.617782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.617905 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.617874    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tnkm\" (UniqueName: \"kubernetes.io/projected/6f1228c3-5493-4572-baef-43101267ddaa-kube-api-access-5tnkm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.617905 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.617902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.719023 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.718980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.719232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.719054    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tnkm\" (UniqueName: \"kubernetes.io/projected/6f1228c3-5493-4572-baef-43101267ddaa-kube-api-access-5tnkm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.719232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.719076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.719232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.719106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.719232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.719124    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1228c3-5493-4572-baef-43101267ddaa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.719232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.719141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.719508 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.719489    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.719554 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.719493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.719598 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.719565    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.721496 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.721472    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1228c3-5493-4572-baef-43101267ddaa-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.721655 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.721638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1228c3-5493-4572-baef-43101267ddaa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.728806 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.728773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tnkm\" (UniqueName: \"kubernetes.io/projected/6f1228c3-5493-4572-baef-43101267ddaa-kube-api-access-5tnkm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf\" (UID: \"6f1228c3-5493-4572-baef-43101267ddaa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:18.812837 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:18.812741    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:19.087896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:19.087853    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"]
Sep 17 14:16:19.090503 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:16:19.090467    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1228c3_5493_4572_baef_43101267ddaa.slice/crio-0507dd618aae3f50e264e178392ac02809a31afaff15c043cecccee2b3944e8c WatchSource:0}: Error finding container 0507dd618aae3f50e264e178392ac02809a31afaff15c043cecccee2b3944e8c: Status 404 returned error can't find the container with id 0507dd618aae3f50e264e178392ac02809a31afaff15c043cecccee2b3944e8c
Sep 17 14:16:19.750867 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:19.750828    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf" event={"ID":"6f1228c3-5493-4572-baef-43101267ddaa","Type":"ContainerStarted","Data":"0507dd618aae3f50e264e178392ac02809a31afaff15c043cecccee2b3944e8c"}
Sep 17 14:16:20.756748 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:20.756713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf" event={"ID":"6f1228c3-5493-4572-baef-43101267ddaa","Type":"ContainerStarted","Data":"efc12715579a3eede4b7a6e81b1cad87f901e1c314defe6f4b2be1e3fe337502"}
Sep 17 14:16:20.775887 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:20.775820    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf" podStartSLOduration=1.603017583 podStartE2EDuration="2.775803826s" totalImagesPullingTime="1.172786243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:20.774284683 +0000 UTC m=+678.365460447" watchObservedRunningTime="2026-09-17 14:16:20.775803826 +0000 UTC m=+678.366979646"
Sep 17 14:16:28.813820 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:28.813771    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:28.814273 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:28.813855    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:28.827081 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:28.827043    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:29.808936 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:29.808906    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf"
Sep 17 14:16:29.879499 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:29.879465    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"]
Sep 17 14:16:29.882987 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:29.882960    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:29.885445 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:29.885416    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 17 14:16:29.896006 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:29.895968    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"]
Sep 17 14:16:30.026464 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.026419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb97r\" (UniqueName: \"kubernetes.io/projected/927b3b41-530b-4681-8f99-fc403b39cc91-kube-api-access-bb97r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.026654 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.026481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/927b3b41-530b-4681-8f99-fc403b39cc91-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.026654 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.026511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.026654 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.026546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.026654 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.026571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.026654 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.026625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.127690 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.127585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.127690 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.127629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.127690 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.127650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.127975 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.127728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb97r\" (UniqueName: \"kubernetes.io/projected/927b3b41-530b-4681-8f99-fc403b39cc91-kube-api-access-bb97r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.127975 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.127768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/927b3b41-530b-4681-8f99-fc403b39cc91-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.127975 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.127798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.128443 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.128410    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.128573 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.128474    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.128573 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.128516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.130381 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.130342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/927b3b41-530b-4681-8f99-fc403b39cc91-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.130519 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.130495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/927b3b41-530b-4681-8f99-fc403b39cc91-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.139220 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.139190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb97r\" (UniqueName: \"kubernetes.io/projected/927b3b41-530b-4681-8f99-fc403b39cc91-kube-api-access-bb97r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7w4m\" (UID: \"927b3b41-530b-4681-8f99-fc403b39cc91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.200987 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.200938    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:30.482018 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.481984    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"]
Sep 17 14:16:30.484104 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:16:30.484068    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927b3b41_530b_4681_8f99_fc403b39cc91.slice/crio-40a9a82487ced0cdd27402c49f2d30f54e539fe7c169da99c122b88ccf1f533d WatchSource:0}: Error finding container 40a9a82487ced0cdd27402c49f2d30f54e539fe7c169da99c122b88ccf1f533d: Status 404 returned error can't find the container with id 40a9a82487ced0cdd27402c49f2d30f54e539fe7c169da99c122b88ccf1f533d
Sep 17 14:16:30.804185 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.804086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m" event={"ID":"927b3b41-530b-4681-8f99-fc403b39cc91","Type":"ContainerStarted","Data":"82343883d21173f8a5530ca1681e8263a8a45a7cdb613b0e9739575554fd6fa5"}
Sep 17 14:16:30.804185 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.804153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m" event={"ID":"927b3b41-530b-4681-8f99-fc403b39cc91","Type":"ContainerStarted","Data":"40a9a82487ced0cdd27402c49f2d30f54e539fe7c169da99c122b88ccf1f533d"}
Sep 17 14:16:30.822951 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:30.822876    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m" podStartSLOduration=1.636553015 podStartE2EDuration="1.82285367s" totalImagesPullingTime="186.300655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:30.821515958 +0000 UTC m=+688.412691719" watchObservedRunningTime="2026-09-17 14:16:30.82285367 +0000 UTC m=+688.414029433"
Sep 17 14:16:40.202137 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:40.202091    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:40.202769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:40.202162    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:40.214820 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:40.214787    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:16:40.866174 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:16:40.866137    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7w4m"
Sep 17 14:17:07.985731 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:07.985628    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"]
Sep 17 14:17:07.989821 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:07.989800    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:07.992658 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:07.992636    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 17 14:17:08.001131 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.001099    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"]
Sep 17 14:17:08.068022 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.067985    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.068207 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.068036    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.068207 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.068059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16b7929-363b-473f-aac0-351c848c1717-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.068207 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.068148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.068323 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.068207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbw88\" (UniqueName: \"kubernetes.io/projected/a16b7929-363b-473f-aac0-351c848c1717-kube-api-access-cbw88\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.068323 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.068257    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.169240 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.169194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.169466 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.169255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.169466 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.169278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16b7929-363b-473f-aac0-351c848c1717-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.169466 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.169330    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.169466 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.169382    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbw88\" (UniqueName: \"kubernetes.io/projected/a16b7929-363b-473f-aac0-351c848c1717-kube-api-access-cbw88\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.169466 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.169411    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.169741 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.169677    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.169741 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.169719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.169848 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.169799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.171611 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.171592    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16b7929-363b-473f-aac0-351c848c1717-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.171865 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.171847    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16b7929-363b-473f-aac0-351c848c1717-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.179478 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.179448    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbw88\" (UniqueName: \"kubernetes.io/projected/a16b7929-363b-473f-aac0-351c848c1717-kube-api-access-cbw88\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72\" (UID: \"a16b7929-363b-473f-aac0-351c848c1717\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.302505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.302416    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:08.581278 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.581249    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"]
Sep 17 14:17:08.583829 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:17:08.583796    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16b7929_363b_473f_aac0_351c848c1717.slice/crio-a00838a09641957eedf6d2e21153b943eff6f5a34d447fa44cf73ef177467c2e WatchSource:0}: Error finding container a00838a09641957eedf6d2e21153b943eff6f5a34d447fa44cf73ef177467c2e: Status 404 returned error can't find the container with id a00838a09641957eedf6d2e21153b943eff6f5a34d447fa44cf73ef177467c2e
Sep 17 14:17:08.585199 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.585181    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:17:08.980174 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.980133    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72" event={"ID":"a16b7929-363b-473f-aac0-351c848c1717","Type":"ContainerStarted","Data":"0e09359691db6257cd6deb903def3e853f4bff289a0430a6546deabbb5ff3381"}
Sep 17 14:17:08.980409 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.980187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72" event={"ID":"a16b7929-363b-473f-aac0-351c848c1717","Type":"ContainerStarted","Data":"a00838a09641957eedf6d2e21153b943eff6f5a34d447fa44cf73ef177467c2e"}
Sep 17 14:17:08.999892 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:08.999830    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72" podStartSLOduration=1.788421081 podStartE2EDuration="1.99981154s" totalImagesPullingTime="211.390459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:08.998814647 +0000 UTC m=+726.589990410" watchObservedRunningTime="2026-09-17 14:17:08.99981154 +0000 UTC m=+726.590987302"
Sep 17 14:17:18.302541 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:18.302510    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:18.303107 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:18.302847    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:18.315440 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:18.315409    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:19.034972 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:19.034941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72"
Sep 17 14:17:32.689263 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.689224    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"]
Sep 17 14:17:32.692476 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.692457    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.695194 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.695165    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 17 14:17:32.703328 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.703299    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"]
Sep 17 14:17:32.796429 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.796384    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.796429 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.796435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxvl\" (UniqueName: \"kubernetes.io/projected/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-kube-api-access-5wxvl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.796663 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.796462    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.796663 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.796489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.796663 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.796559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.796663 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.796585    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.897690 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.897644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.897690 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.897698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.897934 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.897740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.897934 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.897766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.897934 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.897812    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.898091 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.897946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxvl\" (UniqueName: \"kubernetes.io/projected/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-kube-api-access-5wxvl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.898148 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.898123    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.898148 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.898140    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.898253 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.898204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.900158 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.900135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.900298 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.900279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:32.906004 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:32.905972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxvl\" (UniqueName: \"kubernetes.io/projected/6bfad0ac-2050-404d-939d-5cf8cbb8da7a-kube-api-access-5wxvl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7\" (UID: \"6bfad0ac-2050-404d-939d-5cf8cbb8da7a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:33.004726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:33.004688    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:33.285484 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:33.285455    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"]
Sep 17 14:17:33.288024 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:17:33.287995    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bfad0ac_2050_404d_939d_5cf8cbb8da7a.slice/crio-386eb5417e154cf7a9a6a748749cf8a7a3f75aeb67355cbe42fc1dd2b5266039 WatchSource:0}: Error finding container 386eb5417e154cf7a9a6a748749cf8a7a3f75aeb67355cbe42fc1dd2b5266039: Status 404 returned error can't find the container with id 386eb5417e154cf7a9a6a748749cf8a7a3f75aeb67355cbe42fc1dd2b5266039
Sep 17 14:17:34.090850 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:34.090813    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7" event={"ID":"6bfad0ac-2050-404d-939d-5cf8cbb8da7a","Type":"ContainerStarted","Data":"46f0ec8c050fb45f14102a49f9e5f4d86e8070562f0f5d87219db778cf9b206b"}
Sep 17 14:17:34.090850 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:34.090856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7" event={"ID":"6bfad0ac-2050-404d-939d-5cf8cbb8da7a","Type":"ContainerStarted","Data":"386eb5417e154cf7a9a6a748749cf8a7a3f75aeb67355cbe42fc1dd2b5266039"}
Sep 17 14:17:34.118400 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:34.118315    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7" podStartSLOduration=1.921724853 podStartE2EDuration="2.118297291s" totalImagesPullingTime="196.572438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:34.116643653 +0000 UTC m=+751.707819414" watchObservedRunningTime="2026-09-17 14:17:34.118297291 +0000 UTC m=+751.709473052"
Sep 17 14:17:43.004854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:43.004802    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:43.005297 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:43.004870    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:43.017387 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:43.017338    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:17:43.142105 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:17:43.142075    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7"
Sep 17 14:18:56.432191 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.432151    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7877-4b40ff"]
Sep 17 14:18:56.434469 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.434451    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7877-4b40ff"
Sep 17 14:18:56.436795 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.436763    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-25mz9\""
Sep 17 14:18:56.440904 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.440878    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7877-4b40ff"]
Sep 17 14:18:56.492073 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.492034    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c5jg\" (UniqueName: \"kubernetes.io/projected/4d9b2bf9-defc-49ee-bbd4-50f219ce51bb-kube-api-access-2c5jg\") pod \"test-curl-7877-4b40ff\" (UID: \"4d9b2bf9-defc-49ee-bbd4-50f219ce51bb\") " pod="opendatahub/test-curl-7877-4b40ff"
Sep 17 14:18:56.563516 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.563478    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"]
Sep 17 14:18:56.565777 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.565759    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.570145 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.570119    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1269b2-serving-cert\""
Sep 17 14:18:56.593761 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.593662    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"]
Sep 17 14:18:56.593761 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.593754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c5jg\" (UniqueName: \"kubernetes.io/projected/4d9b2bf9-defc-49ee-bbd4-50f219ce51bb-kube-api-access-2c5jg\") pod \"test-curl-7877-4b40ff\" (UID: \"4d9b2bf9-defc-49ee-bbd4-50f219ce51bb\") " pod="opendatahub/test-curl-7877-4b40ff"
Sep 17 14:18:56.597380 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.594192    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.597380 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.594288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-metrics-certs\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.597380 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.594373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87254986-487e-4ac8-8c31-9ba553408fea-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.597380 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.594408    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8msv\" (UniqueName: \"kubernetes.io/projected/87254986-487e-4ac8-8c31-9ba553408fea-kube-api-access-g8msv\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.616597 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.616564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c5jg\" (UniqueName: \"kubernetes.io/projected/4d9b2bf9-defc-49ee-bbd4-50f219ce51bb-kube-api-access-2c5jg\") pod \"test-curl-7877-4b40ff\" (UID: \"4d9b2bf9-defc-49ee-bbd4-50f219ce51bb\") " pod="opendatahub/test-curl-7877-4b40ff"
Sep 17 14:18:56.695176 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.695081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87254986-487e-4ac8-8c31-9ba553408fea-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.695176 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.695122    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8msv\" (UniqueName: \"kubernetes.io/projected/87254986-487e-4ac8-8c31-9ba553408fea-kube-api-access-g8msv\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.695383 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.695182    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.695383 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.695217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-metrics-certs\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.695383 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:18:56.695327    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-serving-cert: secret "maas-api-e2e-worker-w0-1269b2-serving-cert" not found
Sep 17 14:18:56.695522 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:18:56.695440    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-maas-api-tls podName:87254986-487e-4ac8-8c31-9ba553408fea nodeName:}" failed. No retries permitted until 2026-09-17 14:18:57.195423292 +0000 UTC m=+834.786599035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-maas-api-tls") pod "maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x" (UID: "87254986-487e-4ac8-8c31-9ba553408fea") : secret "maas-api-e2e-worker-w0-1269b2-serving-cert" not found
Sep 17 14:18:56.695892 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.695871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87254986-487e-4ac8-8c31-9ba553408fea-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.697771 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.697747    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-metrics-certs\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.705109 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.705080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8msv\" (UniqueName: \"kubernetes.io/projected/87254986-487e-4ac8-8c31-9ba553408fea-kube-api-access-g8msv\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:56.745747 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:56.745706    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7877-4b40ff"
Sep 17 14:18:57.018872 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.018793    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7877-4b40ff"]
Sep 17 14:18:57.021415 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:18:57.021384    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d9b2bf9_defc_49ee_bbd4_50f219ce51bb.slice/crio-e7493ebda253ad0de406506dddfad0482bb287d8fc175bbeaed0816899e8d9e6 WatchSource:0}: Error finding container e7493ebda253ad0de406506dddfad0482bb287d8fc175bbeaed0816899e8d9e6: Status 404 returned error can't find the container with id e7493ebda253ad0de406506dddfad0482bb287d8fc175bbeaed0816899e8d9e6
Sep 17 14:18:57.200786 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.200745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:57.203305 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.203277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:57.223312 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.223268    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"]
Sep 17 14:18:57.226817 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.226786    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:18:57.229254 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.229225    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1269b2\""
Sep 17 14:18:57.238602 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.238571    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"]
Sep 17 14:18:57.302220 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.302124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhgx8\" (UniqueName: \"kubernetes.io/projected/ad04203b-e2ad-4d57-bb53-46c43dfbc242-kube-api-access-bhgx8\") pod \"payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs\" (UID: \"ad04203b-e2ad-4d57-bb53-46c43dfbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:18:57.302220 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.302163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad04203b-e2ad-4d57-bb53-46c43dfbc242-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs\" (UID: \"ad04203b-e2ad-4d57-bb53-46c43dfbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:18:57.403377 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.403322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhgx8\" (UniqueName: \"kubernetes.io/projected/ad04203b-e2ad-4d57-bb53-46c43dfbc242-kube-api-access-bhgx8\") pod \"payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs\" (UID: \"ad04203b-e2ad-4d57-bb53-46c43dfbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:18:57.403582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.403425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad04203b-e2ad-4d57-bb53-46c43dfbc242-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs\" (UID: \"ad04203b-e2ad-4d57-bb53-46c43dfbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:18:57.404138 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.404116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad04203b-e2ad-4d57-bb53-46c43dfbc242-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs\" (UID: \"ad04203b-e2ad-4d57-bb53-46c43dfbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:18:57.414601 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.414568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhgx8\" (UniqueName: \"kubernetes.io/projected/ad04203b-e2ad-4d57-bb53-46c43dfbc242-kube-api-access-bhgx8\") pod \"payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs\" (UID: \"ad04203b-e2ad-4d57-bb53-46c43dfbc242\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:18:57.475841 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.475809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:57.481085 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.481054    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7877-4b40ff" event={"ID":"4d9b2bf9-defc-49ee-bbd4-50f219ce51bb","Type":"ContainerStarted","Data":"e7493ebda253ad0de406506dddfad0482bb287d8fc175bbeaed0816899e8d9e6"}
Sep 17 14:18:57.539582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.539260    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:18:57.751536 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.751511    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"]
Sep 17 14:18:57.753496 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:18:57.753458    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87254986_487e_4ac8_8c31_9ba553408fea.slice/crio-7bf0be010c735f191d13c645e7b4465f81ba9308058db962151d6cb0138b0521 WatchSource:0}: Error finding container 7bf0be010c735f191d13c645e7b4465f81ba9308058db962151d6cb0138b0521: Status 404 returned error can't find the container with id 7bf0be010c735f191d13c645e7b4465f81ba9308058db962151d6cb0138b0521
Sep 17 14:18:57.818954 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:57.818862    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"]
Sep 17 14:18:57.822003 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:18:57.821974    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad04203b_e2ad_4d57_bb53_46c43dfbc242.slice/crio-25a54340af290ffbd7c395695393bd241204ca4426a5579a9d4269707f7c22bd WatchSource:0}: Error finding container 25a54340af290ffbd7c395695393bd241204ca4426a5579a9d4269707f7c22bd: Status 404 returned error can't find the container with id 25a54340af290ffbd7c395695393bd241204ca4426a5579a9d4269707f7c22bd
Sep 17 14:18:58.256330 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.256294    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"]
Sep 17 14:18:58.259662 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.259640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:18:58.262263 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.262231    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-fcd9f3\""
Sep 17 14:18:58.262448 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.262277    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-fcd9f3-dockercfg-q4nhc\""
Sep 17 14:18:58.279188 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.279151    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"]
Sep 17 14:18:58.312706 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.312669    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c78eaec-b277-45e8-8518-63889f408020-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf\" (UID: \"6c78eaec-b277-45e8-8518-63889f408020\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:18:58.312888 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.312734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grl2j\" (UniqueName: \"kubernetes.io/projected/6c78eaec-b277-45e8-8518-63889f408020-kube-api-access-grl2j\") pod \"payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf\" (UID: \"6c78eaec-b277-45e8-8518-63889f408020\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:18:58.413811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.413767    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grl2j\" (UniqueName: \"kubernetes.io/projected/6c78eaec-b277-45e8-8518-63889f408020-kube-api-access-grl2j\") pod \"payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf\" (UID: \"6c78eaec-b277-45e8-8518-63889f408020\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:18:58.413982 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.413904    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c78eaec-b277-45e8-8518-63889f408020-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf\" (UID: \"6c78eaec-b277-45e8-8518-63889f408020\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:18:58.414619 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.414598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c78eaec-b277-45e8-8518-63889f408020-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf\" (UID: \"6c78eaec-b277-45e8-8518-63889f408020\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:18:58.422768 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.422728    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grl2j\" (UniqueName: \"kubernetes.io/projected/6c78eaec-b277-45e8-8518-63889f408020-kube-api-access-grl2j\") pod \"payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf\" (UID: \"6c78eaec-b277-45e8-8518-63889f408020\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:18:58.489311 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.489267    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs" event={"ID":"ad04203b-e2ad-4d57-bb53-46c43dfbc242","Type":"ContainerStarted","Data":"61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8"}
Sep 17 14:18:58.489795 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.489339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs" event={"ID":"ad04203b-e2ad-4d57-bb53-46c43dfbc242","Type":"ContainerStarted","Data":"25a54340af290ffbd7c395695393bd241204ca4426a5579a9d4269707f7c22bd"}
Sep 17 14:18:58.489795 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.489385    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:18:58.491239 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.491212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x" event={"ID":"87254986-487e-4ac8-8c31-9ba553408fea","Type":"ContainerStarted","Data":"b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba"}
Sep 17 14:18:58.491397 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.491261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x" event={"ID":"87254986-487e-4ac8-8c31-9ba553408fea","Type":"ContainerStarted","Data":"7bf0be010c735f191d13c645e7b4465f81ba9308058db962151d6cb0138b0521"}
Sep 17 14:18:58.491397 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.491344    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:18:58.509683 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.509563    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs" podStartSLOduration=1.509545502 podStartE2EDuration="1.509545502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:58.508339524 +0000 UTC m=+836.099515288" watchObservedRunningTime="2026-09-17 14:18:58.509545502 +0000 UTC m=+836.100721258"
Sep 17 14:18:58.517768 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.517731    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"]
Sep 17 14:18:58.521491 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.521465    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.523939 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.523910    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-181b16-serving-cert\""
Sep 17 14:18:58.532986 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.532955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"]
Sep 17 14:18:58.534551 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.534493    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x" podStartSLOduration=1.906613468 podStartE2EDuration="2.534472071s" totalImagesPullingTime="627.858603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:58.533948788 +0000 UTC m=+836.125124551" watchObservedRunningTime="2026-09-17 14:18:58.534472071 +0000 UTC m=+836.125647834"
Sep 17 14:18:58.571638 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.571605    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:18:58.616532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.616464    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-metrics-certs\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.616743 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.616558    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hhg\" (UniqueName: \"kubernetes.io/projected/a3334318-e29e-48ea-9c05-b36424bef375-kube-api-access-m8hhg\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.616743 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.616671    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3334318-e29e-48ea-9c05-b36424bef375-openshift-service-ca\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.616743 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.616703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-maas-api-tls\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.717467 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.717426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3334318-e29e-48ea-9c05-b36424bef375-openshift-service-ca\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.717467 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.717468    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-maas-api-tls\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.717467 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.717530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-metrics-certs\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.717897 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.717567    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hhg\" (UniqueName: \"kubernetes.io/projected/a3334318-e29e-48ea-9c05-b36424bef375-kube-api-access-m8hhg\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.718291 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.718267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3334318-e29e-48ea-9c05-b36424bef375-openshift-service-ca\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.720106 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.720085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-metrics-certs\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.720205 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.720122    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-maas-api-tls\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.726160 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.726128    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hhg\" (UniqueName: \"kubernetes.io/projected/a3334318-e29e-48ea-9c05-b36424bef375-kube-api-access-m8hhg\") pod \"maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.833682 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.833595    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:18:58.853114 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:58.853077    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"]
Sep 17 14:18:58.854926 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:18:58.854389    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c78eaec_b277_45e8_8518_63889f408020.slice/crio-062d8db42d650019d58da75b222356cf5b45f5516bf731872998002d540ba726 WatchSource:0}: Error finding container 062d8db42d650019d58da75b222356cf5b45f5516bf731872998002d540ba726: Status 404 returned error can't find the container with id 062d8db42d650019d58da75b222356cf5b45f5516bf731872998002d540ba726
Sep 17 14:18:59.128314 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:59.128221    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"]
Sep 17 14:18:59.131913 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:18:59.131858    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3334318_e29e_48ea_9c05_b36424bef375.slice/crio-e62cd6ed32ac7e9aa6ae8470cb84871dff2eba8a5221b7bedf085263b72012d0 WatchSource:0}: Error finding container e62cd6ed32ac7e9aa6ae8470cb84871dff2eba8a5221b7bedf085263b72012d0: Status 404 returned error can't find the container with id e62cd6ed32ac7e9aa6ae8470cb84871dff2eba8a5221b7bedf085263b72012d0
Sep 17 14:18:59.498090 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:59.498050    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6" event={"ID":"a3334318-e29e-48ea-9c05-b36424bef375","Type":"ContainerStarted","Data":"e62cd6ed32ac7e9aa6ae8470cb84871dff2eba8a5221b7bedf085263b72012d0"}
Sep 17 14:18:59.500062 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:59.500024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf" event={"ID":"6c78eaec-b277-45e8-8518-63889f408020","Type":"ContainerStarted","Data":"c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5"}
Sep 17 14:18:59.500226 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:59.500079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf" event={"ID":"6c78eaec-b277-45e8-8518-63889f408020","Type":"ContainerStarted","Data":"062d8db42d650019d58da75b222356cf5b45f5516bf731872998002d540ba726"}
Sep 17 14:18:59.500504 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:59.500480    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:18:59.523268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:18:59.523202    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf" podStartSLOduration=1.5231819469999999 podStartE2EDuration="1.523181947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:59.520189272 +0000 UTC m=+837.111365051" watchObservedRunningTime="2026-09-17 14:18:59.523181947 +0000 UTC m=+837.114357709"
Sep 17 14:19:00.509859 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:00.509818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6" event={"ID":"a3334318-e29e-48ea-9c05-b36424bef375","Type":"ContainerStarted","Data":"a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323"}
Sep 17 14:19:00.544600 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:00.544532    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6" podStartSLOduration=2.134049607 podStartE2EDuration="2.544505814s" totalImagesPullingTime="410.456207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:00.542210745 +0000 UTC m=+838.133386529" watchObservedRunningTime="2026-09-17 14:19:00.544505814 +0000 UTC m=+838.135681576"
Sep 17 14:19:01.517293 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:01.517250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7877-4b40ff" event={"ID":"4d9b2bf9-defc-49ee-bbd4-50f219ce51bb","Type":"ContainerStarted","Data":"0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938"}
Sep 17 14:19:01.517732 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:01.517555    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:19:01.541076 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:01.541018    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7877-4b40ff" podStartSLOduration=1.980141092 podStartE2EDuration="5.541001125s" totalImagesPullingTime="3.560860033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:01.539055512 +0000 UTC m=+839.130231273" watchObservedRunningTime="2026-09-17 14:19:01.541001125 +0000 UTC m=+839.132176885"
Sep 17 14:19:02.155189 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:02.155150    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7877-4b40ff"]
Sep 17 14:19:02.159853 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:02.159817    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7877-4b40ff"]
Sep 17 14:19:02.522406 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:19:02.522374    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7877-4b40ff" container="test-curl-7877-4b40ff"
Sep 17 14:19:03.527898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:03.527856    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7877-4b40ff" podUID="4d9b2bf9-defc-49ee-bbd4-50f219ce51bb" containerName="test-curl-7877-4b40ff" containerID="cri-o://0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938" gracePeriod=2
Sep 17 14:19:04.506245 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:04.506206    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:19:05.714940 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:05.714912    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7877-4b40ff"
Sep 17 14:19:05.789494 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:05.789389    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c5jg\" (UniqueName: \"kubernetes.io/projected/4d9b2bf9-defc-49ee-bbd4-50f219ce51bb-kube-api-access-2c5jg\") pod \"4d9b2bf9-defc-49ee-bbd4-50f219ce51bb\" (UID: \"4d9b2bf9-defc-49ee-bbd4-50f219ce51bb\") "
Sep 17 14:19:05.791583 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:05.791552    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9b2bf9-defc-49ee-bbd4-50f219ce51bb-kube-api-access-2c5jg" (OuterVolumeSpecName: "kube-api-access-2c5jg") pod "4d9b2bf9-defc-49ee-bbd4-50f219ce51bb" (UID: "4d9b2bf9-defc-49ee-bbd4-50f219ce51bb"). InnerVolumeSpecName "kube-api-access-2c5jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:19:05.891059 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:05.891014    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c5jg\" (UniqueName: \"kubernetes.io/projected/4d9b2bf9-defc-49ee-bbd4-50f219ce51bb-kube-api-access-2c5jg\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:19:06.543818 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:06.543777    2560 generic.go:356] "Generic (PLEG): container finished" podID="4d9b2bf9-defc-49ee-bbd4-50f219ce51bb" containerID="0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938" exitCode=137
Sep 17 14:19:06.543975 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:06.543832    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7877-4b40ff"
Sep 17 14:19:06.543975 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:06.543849    2560 scope.go:117] "RemoveContainer" containerID="0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938"
Sep 17 14:19:06.552976 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:06.552957    2560 scope.go:117] "RemoveContainer" containerID="0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938"
Sep 17 14:19:06.553325 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:19:06.553303    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938\": container with ID starting with 0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938 not found: ID does not exist" containerID="0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938"
Sep 17 14:19:06.553394 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:06.553336    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938"} err="failed to get container status \"0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938\": rpc error: code = NotFound desc = could not find container \"0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938\": container with ID starting with 0e49fdec0b2ee26b5ab568d513eb32039e562e0181a38217d7860e07bed00938 not found: ID does not exist"
Sep 17 14:19:06.554908 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:19:06.554880    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7877-4b40ff\" is forbidden: User \"system:node:ip-10-0-131-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-146.ec2.internal' and this object" podUID="4d9b2bf9-defc-49ee-bbd4-50f219ce51bb" pod="opendatahub/test-curl-7877-4b40ff"
Sep 17 14:19:06.965700 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:06.965608    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9b2bf9-defc-49ee-bbd4-50f219ce51bb" path="/var/lib/kubelet/pods/4d9b2bf9-defc-49ee-bbd4-50f219ce51bb/volumes"
Sep 17 14:19:07.528863 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:07.528830    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:19:08.498220 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.498182    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"]
Sep 17 14:19:08.498933 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.498908    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9b2bf9-defc-49ee-bbd4-50f219ce51bb" containerName="test-curl-7877-4b40ff"
Sep 17 14:19:08.499083 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.499070    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9b2bf9-defc-49ee-bbd4-50f219ce51bb" containerName="test-curl-7877-4b40ff"
Sep 17 14:19:08.499289 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.499276    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9b2bf9-defc-49ee-bbd4-50f219ce51bb" containerName="test-curl-7877-4b40ff"
Sep 17 14:19:08.504376 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.504332    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.506711 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.506672    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ccd499b1-serving-cert\""
Sep 17 14:19:08.512897 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.512863    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"]
Sep 17 14:19:08.614550 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.614500    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-metrics-certs\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.614754 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.614582    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-maas-api-tls\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.614754 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.614608    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67949df5-b133-4231-9450-adec5e2abaf0-openshift-service-ca\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.614754 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.614681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct2j6\" (UniqueName: \"kubernetes.io/projected/67949df5-b133-4231-9450-adec5e2abaf0-kube-api-access-ct2j6\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.715488 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.715442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-metrics-certs\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.715676 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.715531    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-maas-api-tls\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.715676 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.715562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67949df5-b133-4231-9450-adec5e2abaf0-openshift-service-ca\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.715676 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.715595    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct2j6\" (UniqueName: \"kubernetes.io/projected/67949df5-b133-4231-9450-adec5e2abaf0-kube-api-access-ct2j6\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.715856 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:19:08.715685    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-serving-cert: secret "maas-api-e2e-ait-ccd499b1-serving-cert" not found
Sep 17 14:19:08.715856 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:19:08.715789    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-maas-api-tls podName:67949df5-b133-4231-9450-adec5e2abaf0 nodeName:}" failed. No retries permitted until 2026-09-17 14:19:09.215770896 +0000 UTC m=+846.806946649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-maas-api-tls") pod "maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4" (UID: "67949df5-b133-4231-9450-adec5e2abaf0") : secret "maas-api-e2e-ait-ccd499b1-serving-cert" not found
Sep 17 14:19:08.716259 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.716237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67949df5-b133-4231-9450-adec5e2abaf0-openshift-service-ca\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.718009 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.717970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-metrics-certs\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:08.726811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:08.726783    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct2j6\" (UniqueName: \"kubernetes.io/projected/67949df5-b133-4231-9450-adec5e2abaf0-kube-api-access-ct2j6\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:09.220111 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.220072    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-maas-api-tls\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:09.222646 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.222615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-maas-api-tls\") pod \"maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:09.281410 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.281348    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"]
Sep 17 14:19:09.285164 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.285138    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:09.287719 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.287691    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ccd499b1\""
Sep 17 14:19:09.287863 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.287740    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ccd499b1-dockercfg-wg48g\""
Sep 17 14:19:09.295645 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.295609    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"]
Sep 17 14:19:09.325037 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.325001    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"]
Sep 17 14:19:09.328545 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.328522    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:09.339923 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.339864    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"]
Sep 17 14:19:09.419469 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.419429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:09.421877 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.421846    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d9zs\" (UniqueName: \"kubernetes.io/projected/9ee37a00-22dc-46ea-8243-0fe4d28f0684-kube-api-access-7d9zs\") pod \"payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh\" (UID: \"9ee37a00-22dc-46ea-8243-0fe4d28f0684\") " pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:09.422100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.422081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82\" (UID: \"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:09.422187 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.422120    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee37a00-22dc-46ea-8243-0fe4d28f0684-plugins-config-volume\") pod \"payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh\" (UID: \"9ee37a00-22dc-46ea-8243-0fe4d28f0684\") " pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:09.422187 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.422171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4kl\" (UniqueName: \"kubernetes.io/projected/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-kube-api-access-kn4kl\") pod \"payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82\" (UID: \"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:09.502259 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.502229    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:19:09.523274 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.523230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d9zs\" (UniqueName: \"kubernetes.io/projected/9ee37a00-22dc-46ea-8243-0fe4d28f0684-kube-api-access-7d9zs\") pod \"payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh\" (UID: \"9ee37a00-22dc-46ea-8243-0fe4d28f0684\") " pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:09.523274 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.523280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82\" (UID: \"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:09.523582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.523412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee37a00-22dc-46ea-8243-0fe4d28f0684-plugins-config-volume\") pod \"payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh\" (UID: \"9ee37a00-22dc-46ea-8243-0fe4d28f0684\") " pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:09.523582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.523504    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4kl\" (UniqueName: \"kubernetes.io/projected/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-kube-api-access-kn4kl\") pod \"payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82\" (UID: \"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:09.524406 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.524325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee37a00-22dc-46ea-8243-0fe4d28f0684-plugins-config-volume\") pod \"payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh\" (UID: \"9ee37a00-22dc-46ea-8243-0fe4d28f0684\") " pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:09.524406 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.524391    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82\" (UID: \"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:09.531869 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.531830    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4kl\" (UniqueName: \"kubernetes.io/projected/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-kube-api-access-kn4kl\") pod \"payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82\" (UID: \"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:09.532758 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.532727    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d9zs\" (UniqueName: \"kubernetes.io/projected/9ee37a00-22dc-46ea-8243-0fe4d28f0684-kube-api-access-7d9zs\") pod \"payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh\" (UID: \"9ee37a00-22dc-46ea-8243-0fe4d28f0684\") " pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:09.597301 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.597261    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:09.639826 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.639780    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:09.700491 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.700458    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"]
Sep 17 14:19:09.703822 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:19:09.703790    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67949df5_b133_4231_9450_adec5e2abaf0.slice/crio-730e02e0d144443f23831ae4eba1bd2dd36d7de4df12edc20a7e1b1cf482c686 WatchSource:0}: Error finding container 730e02e0d144443f23831ae4eba1bd2dd36d7de4df12edc20a7e1b1cf482c686: Status 404 returned error can't find the container with id 730e02e0d144443f23831ae4eba1bd2dd36d7de4df12edc20a7e1b1cf482c686
Sep 17 14:19:09.882833 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.882807    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"]
Sep 17 14:19:09.884910 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:19:09.884878    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode362c0c0_4b7a_49f7_9fc7_8a38e59fe2dd.slice/crio-c5a1a4183b5a8659936c32408845984b334511fc9269bbffbc6e9abedfccf921 WatchSource:0}: Error finding container c5a1a4183b5a8659936c32408845984b334511fc9269bbffbc6e9abedfccf921: Status 404 returned error can't find the container with id c5a1a4183b5a8659936c32408845984b334511fc9269bbffbc6e9abedfccf921
Sep 17 14:19:09.932120 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:09.932096    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"]
Sep 17 14:19:09.940635 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:19:09.940602    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee37a00_22dc_46ea_8243_0fe4d28f0684.slice/crio-f3550865143463109cc986cb6a39ea91ed0b331958bb3ea4cd38c1dd63996fec WatchSource:0}: Error finding container f3550865143463109cc986cb6a39ea91ed0b331958bb3ea4cd38c1dd63996fec: Status 404 returned error can't find the container with id f3550865143463109cc986cb6a39ea91ed0b331958bb3ea4cd38c1dd63996fec
Sep 17 14:19:10.512182 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.512148    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:19:10.570881 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.570840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4" event={"ID":"67949df5-b133-4231-9450-adec5e2abaf0","Type":"ContainerStarted","Data":"524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec"}
Sep 17 14:19:10.571085 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.570904    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4" event={"ID":"67949df5-b133-4231-9450-adec5e2abaf0","Type":"ContainerStarted","Data":"730e02e0d144443f23831ae4eba1bd2dd36d7de4df12edc20a7e1b1cf482c686"}
Sep 17 14:19:10.571085 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.571005    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:10.572895 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.572804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh" event={"ID":"9ee37a00-22dc-46ea-8243-0fe4d28f0684","Type":"ContainerStarted","Data":"6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad"}
Sep 17 14:19:10.572895 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.572870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh" event={"ID":"9ee37a00-22dc-46ea-8243-0fe4d28f0684","Type":"ContainerStarted","Data":"f3550865143463109cc986cb6a39ea91ed0b331958bb3ea4cd38c1dd63996fec"}
Sep 17 14:19:10.573111 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.572902    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:10.574792 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.574759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82" event={"ID":"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd","Type":"ContainerStarted","Data":"7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351"}
Sep 17 14:19:10.574943 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.574814    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82" event={"ID":"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd","Type":"ContainerStarted","Data":"c5a1a4183b5a8659936c32408845984b334511fc9269bbffbc6e9abedfccf921"}
Sep 17 14:19:10.574943 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.574921    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:10.589982 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.589918    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4" podStartSLOduration=2.2494435680000002 podStartE2EDuration="2.589897392s" totalImagesPullingTime="340.453824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:10.587956403 +0000 UTC m=+848.179132179" watchObservedRunningTime="2026-09-17 14:19:10.589897392 +0000 UTC m=+848.181073155"
Sep 17 14:19:10.604272 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.604205    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh" podStartSLOduration=1.604183678 podStartE2EDuration="1.604183678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:10.603904485 +0000 UTC m=+848.195080248" watchObservedRunningTime="2026-09-17 14:19:10.604183678 +0000 UTC m=+848.195359438"
Sep 17 14:19:10.620553 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:10.620483    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82" podStartSLOduration=1.620460013 podStartE2EDuration="1.620460013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:10.620034995 +0000 UTC m=+848.211210758" watchObservedRunningTime="2026-09-17 14:19:10.620460013 +0000 UTC m=+848.211635778"
Sep 17 14:19:11.248043 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.248007    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"]
Sep 17 14:19:11.251959 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.251930    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:19:11.254563 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.254528    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3f6f32\""
Sep 17 14:19:11.254710 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.254617    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3f6f32-dockercfg-9r7dw\""
Sep 17 14:19:11.264973 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.264941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"]
Sep 17 14:19:11.343508 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.343456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7tz\" (UniqueName: \"kubernetes.io/projected/f227571c-15b2-445b-80ff-9aa07b9b7a13-kube-api-access-zm7tz\") pod \"payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225\" (UID: \"f227571c-15b2-445b-80ff-9aa07b9b7a13\") " pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:19:11.343736 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.343609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f227571c-15b2-445b-80ff-9aa07b9b7a13-plugins-config-volume\") pod \"payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225\" (UID: \"f227571c-15b2-445b-80ff-9aa07b9b7a13\") " pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:19:11.444728 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.444659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7tz\" (UniqueName: \"kubernetes.io/projected/f227571c-15b2-445b-80ff-9aa07b9b7a13-kube-api-access-zm7tz\") pod \"payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225\" (UID: \"f227571c-15b2-445b-80ff-9aa07b9b7a13\") " pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:19:11.444927 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.444811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f227571c-15b2-445b-80ff-9aa07b9b7a13-plugins-config-volume\") pod \"payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225\" (UID: \"f227571c-15b2-445b-80ff-9aa07b9b7a13\") " pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:19:11.445818 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.445792    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f227571c-15b2-445b-80ff-9aa07b9b7a13-plugins-config-volume\") pod \"payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225\" (UID: \"f227571c-15b2-445b-80ff-9aa07b9b7a13\") " pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:19:11.454184 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.454155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7tz\" (UniqueName: \"kubernetes.io/projected/f227571c-15b2-445b-80ff-9aa07b9b7a13-kube-api-access-zm7tz\") pod \"payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225\" (UID: \"f227571c-15b2-445b-80ff-9aa07b9b7a13\") " pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:19:11.567736 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.567632    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:19:11.848920 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:11.848883    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"]
Sep 17 14:19:11.851106 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:19:11.851080    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf227571c_15b2_445b_80ff_9aa07b9b7a13.slice/crio-16430a149a119d2d0329bdb80e5d032550295ed09452f350c096d1055762dccf WatchSource:0}: Error finding container 16430a149a119d2d0329bdb80e5d032550295ed09452f350c096d1055762dccf: Status 404 returned error can't find the container with id 16430a149a119d2d0329bdb80e5d032550295ed09452f350c096d1055762dccf
Sep 17 14:19:12.592560 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:12.592525    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225" event={"ID":"f227571c-15b2-445b-80ff-9aa07b9b7a13","Type":"ContainerStarted","Data":"ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11"}
Sep 17 14:19:12.592982 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:12.592588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225" event={"ID":"f227571c-15b2-445b-80ff-9aa07b9b7a13","Type":"ContainerStarted","Data":"16430a149a119d2d0329bdb80e5d032550295ed09452f350c096d1055762dccf"}
Sep 17 14:19:12.592982 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:12.592645    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:19:12.634906 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:12.634837    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225" podStartSLOduration=1.634817341 podStartE2EDuration="1.634817341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:12.632913871 +0000 UTC m=+850.224089634" watchObservedRunningTime="2026-09-17 14:19:12.634817341 +0000 UTC m=+850.225993102"
Sep 17 14:19:16.586850 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:16.586818    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:21.583566 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:21.583538    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:21.584016 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:21.583604    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:23.600562 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:23.600533    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:19:32.006171 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.006124    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"]
Sep 17 14:19:32.009808 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.009780    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.012660 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.012634    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1269b2\"/\"openshift-service-ca.crt\""
Sep 17 14:19:32.013911 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.013889    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1269b2\"/\"simulator-w0-1269b2-kserve-self-signed-certs\""
Sep 17 14:19:32.014075 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.013917    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1269b2\"/\"default-dockercfg-dsxwt\""
Sep 17 14:19:32.014075 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.013947    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1269b2\"/\"kube-root-ca.crt\""
Sep 17 14:19:32.019187 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.019149    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"]
Sep 17 14:19:32.037121 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.037083    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-model-cache\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.037318 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.037182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-home\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.037318 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.037223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr4d9\" (UniqueName: \"kubernetes.io/projected/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-kube-api-access-kr4d9\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.037318 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.037267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tmp-dir\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.037318 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.037297    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-dshm\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.037529 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.037376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tls-certs\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.055232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.055191    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"]
Sep 17 14:19:32.055472 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.055445    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4" podUID="67949df5-b133-4231-9450-adec5e2abaf0" containerName="maas-api" containerID="cri-o://524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec" gracePeriod=30
Sep 17 14:19:32.117056 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.117021    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"]
Sep 17 14:19:32.117270 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.117244    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh" podUID="9ee37a00-22dc-46ea-8243-0fe4d28f0684" containerName="payload-processing" containerID="cri-o://6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad" gracePeriod=30
Sep 17 14:19:32.124227 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.124186    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"]
Sep 17 14:19:32.124512 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.124481    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82" podUID="e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd" containerName="payload-pre-processing" containerID="cri-o://7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351" gracePeriod=30
Sep 17 14:19:32.139397 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.138279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-model-cache\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.139729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.139695    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-home\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.139844 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.139792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr4d9\" (UniqueName: \"kubernetes.io/projected/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-kube-api-access-kr4d9\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.139908 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.139842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tmp-dir\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.139908 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.139870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-dshm\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.139908 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.139894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tls-certs\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.140073 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.140010    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-model-cache\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.141486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.140565    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-home\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.141486 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.140634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tmp-dir\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.143236 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.143188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-dshm\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.143724 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.143702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tls-certs\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.148990 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.148961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr4d9\" (UniqueName: \"kubernetes.io/projected/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-kube-api-access-kr4d9\") pod \"simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") " pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.320380 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.320321    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:32.430383 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.430334    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:32.439213 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.439180    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:32.443735 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.443706    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d9zs\" (UniqueName: \"kubernetes.io/projected/9ee37a00-22dc-46ea-8243-0fe4d28f0684-kube-api-access-7d9zs\") pod \"9ee37a00-22dc-46ea-8243-0fe4d28f0684\" (UID: \"9ee37a00-22dc-46ea-8243-0fe4d28f0684\") "
Sep 17 14:19:32.443920 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.443751    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee37a00-22dc-46ea-8243-0fe4d28f0684-plugins-config-volume\") pod \"9ee37a00-22dc-46ea-8243-0fe4d28f0684\" (UID: \"9ee37a00-22dc-46ea-8243-0fe4d28f0684\") "
Sep 17 14:19:32.444200 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.444168    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee37a00-22dc-46ea-8243-0fe4d28f0684-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ee37a00-22dc-46ea-8243-0fe4d28f0684" (UID: "9ee37a00-22dc-46ea-8243-0fe4d28f0684"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:19:32.446930 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.446897    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee37a00-22dc-46ea-8243-0fe4d28f0684-kube-api-access-7d9zs" (OuterVolumeSpecName: "kube-api-access-7d9zs") pod "9ee37a00-22dc-46ea-8243-0fe4d28f0684" (UID: "9ee37a00-22dc-46ea-8243-0fe4d28f0684"). InnerVolumeSpecName "kube-api-access-7d9zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:19:32.466279 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.466238    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:32.545457 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.545372    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-maas-api-tls\") pod \"67949df5-b133-4231-9450-adec5e2abaf0\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") "
Sep 17 14:19:32.545457 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.545447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn4kl\" (UniqueName: \"kubernetes.io/projected/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-kube-api-access-kn4kl\") pod \"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd\" (UID: \"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd\") "
Sep 17 14:19:32.546036 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.546004    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd" (UID: "e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:19:32.546141 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.546118    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-plugins-config-volume\") pod \"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd\" (UID: \"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd\") "
Sep 17 14:19:32.546268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.546256    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67949df5-b133-4231-9450-adec5e2abaf0-openshift-service-ca\") pod \"67949df5-b133-4231-9450-adec5e2abaf0\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") "
Sep 17 14:19:32.547163 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.546567    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct2j6\" (UniqueName: \"kubernetes.io/projected/67949df5-b133-4231-9450-adec5e2abaf0-kube-api-access-ct2j6\") pod \"67949df5-b133-4231-9450-adec5e2abaf0\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") "
Sep 17 14:19:32.547163 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.546693    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-metrics-certs\") pod \"67949df5-b133-4231-9450-adec5e2abaf0\" (UID: \"67949df5-b133-4231-9450-adec5e2abaf0\") "
Sep 17 14:19:32.547163 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.546820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67949df5-b133-4231-9450-adec5e2abaf0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67949df5-b133-4231-9450-adec5e2abaf0" (UID: "67949df5-b133-4231-9450-adec5e2abaf0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:19:32.547163 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.547069    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:19:32.547163 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.547089    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67949df5-b133-4231-9450-adec5e2abaf0-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:19:32.547163 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.547106    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d9zs\" (UniqueName: \"kubernetes.io/projected/9ee37a00-22dc-46ea-8243-0fe4d28f0684-kube-api-access-7d9zs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:19:32.547163 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.547120    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee37a00-22dc-46ea-8243-0fe4d28f0684-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:19:32.548857 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.548825    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67949df5-b133-4231-9450-adec5e2abaf0" (UID: "67949df5-b133-4231-9450-adec5e2abaf0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:19:32.548962 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.548827    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-kube-api-access-kn4kl" (OuterVolumeSpecName: "kube-api-access-kn4kl") pod "e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd" (UID: "e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd"). InnerVolumeSpecName "kube-api-access-kn4kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:19:32.549076 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.549048    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67949df5-b133-4231-9450-adec5e2abaf0-kube-api-access-ct2j6" (OuterVolumeSpecName: "kube-api-access-ct2j6") pod "67949df5-b133-4231-9450-adec5e2abaf0" (UID: "67949df5-b133-4231-9450-adec5e2abaf0"). InnerVolumeSpecName "kube-api-access-ct2j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:19:32.549388 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.549339    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "67949df5-b133-4231-9450-adec5e2abaf0" (UID: "67949df5-b133-4231-9450-adec5e2abaf0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:19:32.613176 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.613147    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"]
Sep 17 14:19:32.648567 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.648536    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:19:32.648567 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.648567    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn4kl\" (UniqueName: \"kubernetes.io/projected/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd-kube-api-access-kn4kl\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:19:32.648763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.648580    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct2j6\" (UniqueName: \"kubernetes.io/projected/67949df5-b133-4231-9450-adec5e2abaf0-kube-api-access-ct2j6\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:19:32.648763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.648592    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67949df5-b133-4231-9450-adec5e2abaf0-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:19:32.700225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.700190    2560 generic.go:356] "Generic (PLEG): container finished" podID="e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd" containerID="7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351" exitCode=0
Sep 17 14:19:32.700450 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.700236    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"
Sep 17 14:19:32.700450 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.700281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82" event={"ID":"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd","Type":"ContainerDied","Data":"7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351"}
Sep 17 14:19:32.700450 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.700349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82" event={"ID":"e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd","Type":"ContainerDied","Data":"c5a1a4183b5a8659936c32408845984b334511fc9269bbffbc6e9abedfccf921"}
Sep 17 14:19:32.700450 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.700392    2560 scope.go:117] "RemoveContainer" containerID="7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351"
Sep 17 14:19:32.702350 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.702313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq" event={"ID":"4843a90f-8f8f-4a04-93a0-1bb9dd20c850","Type":"ContainerStarted","Data":"ee08be6e04f1b982e6f16c3bdcd82cf1f74aa7bf7c9e75241282db6b5aa385fd"}
Sep 17 14:19:32.702492 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.702389    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq" event={"ID":"4843a90f-8f8f-4a04-93a0-1bb9dd20c850","Type":"ContainerStarted","Data":"c52dff85cf8381c154614f1e109ca366ee46bc82c15123c0a60611650c41c858"}
Sep 17 14:19:32.704288 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.704257    2560 generic.go:356] "Generic (PLEG): container finished" podID="67949df5-b133-4231-9450-adec5e2abaf0" containerID="524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec" exitCode=0
Sep 17 14:19:32.704441 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.704291    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4" event={"ID":"67949df5-b133-4231-9450-adec5e2abaf0","Type":"ContainerDied","Data":"524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec"}
Sep 17 14:19:32.704441 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.704306    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"
Sep 17 14:19:32.704441 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.704352    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4" event={"ID":"67949df5-b133-4231-9450-adec5e2abaf0","Type":"ContainerDied","Data":"730e02e0d144443f23831ae4eba1bd2dd36d7de4df12edc20a7e1b1cf482c686"}
Sep 17 14:19:32.706276 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.706246    2560 generic.go:356] "Generic (PLEG): container finished" podID="9ee37a00-22dc-46ea-8243-0fe4d28f0684" containerID="6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad" exitCode=0
Sep 17 14:19:32.706473 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.706287    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh" event={"ID":"9ee37a00-22dc-46ea-8243-0fe4d28f0684","Type":"ContainerDied","Data":"6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad"}
Sep 17 14:19:32.706473 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.706306    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"
Sep 17 14:19:32.706473 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.706330    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh" event={"ID":"9ee37a00-22dc-46ea-8243-0fe4d28f0684","Type":"ContainerDied","Data":"f3550865143463109cc986cb6a39ea91ed0b331958bb3ea4cd38c1dd63996fec"}
Sep 17 14:19:32.718869 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.718843    2560 scope.go:117] "RemoveContainer" containerID="7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351"
Sep 17 14:19:32.719299 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:19:32.719275    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351\": container with ID starting with 7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351 not found: ID does not exist" containerID="7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351"
Sep 17 14:19:32.719435 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.719311    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351"} err="failed to get container status \"7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351\": rpc error: code = NotFound desc = could not find container \"7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351\": container with ID starting with 7b997dfb5acdeb452952b385cdadb1558690d803de27d0cb260051bbdc270351 not found: ID does not exist"
Sep 17 14:19:32.719435 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.719338    2560 scope.go:117] "RemoveContainer" containerID="524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec"
Sep 17 14:19:32.725925 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.725871    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq" podStartSLOduration=1.725849349 podStartE2EDuration="1.725849349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:32.723900804 +0000 UTC m=+870.315076565" watchObservedRunningTime="2026-09-17 14:19:32.725849349 +0000 UTC m=+870.317025112"
Sep 17 14:19:32.730536 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.730517    2560 scope.go:117] "RemoveContainer" containerID="524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec"
Sep 17 14:19:32.731011 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:19:32.730982    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec\": container with ID starting with 524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec not found: ID does not exist" containerID="524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec"
Sep 17 14:19:32.731118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.731019    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec"} err="failed to get container status \"524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec\": rpc error: code = NotFound desc = could not find container \"524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec\": container with ID starting with 524cdb7dc053847a2c3056a91fe59cb1274c84aaa619288a4706edea943510ec not found: ID does not exist"
Sep 17 14:19:32.731118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.731038    2560 scope.go:117] "RemoveContainer" containerID="6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad"
Sep 17 14:19:32.741621 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.741584    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"]
Sep 17 14:19:32.742055 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.742034    2560 scope.go:117] "RemoveContainer" containerID="6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad"
Sep 17 14:19:32.742526 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:19:32.742498    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad\": container with ID starting with 6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad not found: ID does not exist" containerID="6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad"
Sep 17 14:19:32.742609 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.742536    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad"} err="failed to get container status \"6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad\": rpc error: code = NotFound desc = could not find container \"6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad\": container with ID starting with 6a278eb829983b2de3c8b30a27b482d70ff1d0061226fc66e22b7f943a78adad not found: ID does not exist"
Sep 17 14:19:32.744152 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.744127    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccd499b1-6db4b5477c-kxqc4"]
Sep 17 14:19:32.760912 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.760879    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"]
Sep 17 14:19:32.765656 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.765622    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccd499b1-84c55ccbc4-lsc82"]
Sep 17 14:19:32.779149 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.778917    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"]
Sep 17 14:19:32.781631 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.781605    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccd499b1-5c7c4cf645-xmjrh"]
Sep 17 14:19:32.965805 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.965708    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67949df5-b133-4231-9450-adec5e2abaf0" path="/var/lib/kubelet/pods/67949df5-b133-4231-9450-adec5e2abaf0/volumes"
Sep 17 14:19:32.966280 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.966228    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ee37a00-22dc-46ea-8243-0fe4d28f0684" path="/var/lib/kubelet/pods/9ee37a00-22dc-46ea-8243-0fe4d28f0684/volumes"
Sep 17 14:19:32.966651 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:32.966632    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd" path="/var/lib/kubelet/pods/e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd/volumes"
Sep 17 14:19:39.500476 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.500434    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"]
Sep 17 14:19:39.501014 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.500997    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd" containerName="payload-pre-processing"
Sep 17 14:19:39.501062 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.501017    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd" containerName="payload-pre-processing"
Sep 17 14:19:39.501062 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.501043    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67949df5-b133-4231-9450-adec5e2abaf0" containerName="maas-api"
Sep 17 14:19:39.501062 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.501050    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="67949df5-b133-4231-9450-adec5e2abaf0" containerName="maas-api"
Sep 17 14:19:39.501168 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.501062    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee37a00-22dc-46ea-8243-0fe4d28f0684" containerName="payload-processing"
Sep 17 14:19:39.501168 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.501068    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee37a00-22dc-46ea-8243-0fe4d28f0684" containerName="payload-processing"
Sep 17 14:19:39.501168 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.501134    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e362c0c0-4b7a-49f7-9fc7-8a38e59fe2dd" containerName="payload-pre-processing"
Sep 17 14:19:39.501168 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.501144    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee37a00-22dc-46ea-8243-0fe4d28f0684" containerName="payload-processing"
Sep 17 14:19:39.501168 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.501151    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="67949df5-b133-4231-9450-adec5e2abaf0" containerName="maas-api"
Sep 17 14:19:39.506560 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.506530    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.509139 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.509110    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9e4372d6-serving-cert\""
Sep 17 14:19:39.517064 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.517027    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"]
Sep 17 14:19:39.621179 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.621135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-metrics-certs\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.621412 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.621274    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-maas-api-tls\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.621412 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.621314    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfa14d1-afe4-497c-9c5e-062e742d53bf-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.621412 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.621337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jjgl\" (UniqueName: \"kubernetes.io/projected/6dfa14d1-afe4-497c-9c5e-062e742d53bf-kube-api-access-6jjgl\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.721970 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.721939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-maas-api-tls\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.722164 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.721981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfa14d1-afe4-497c-9c5e-062e742d53bf-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.722164 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.722008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jjgl\" (UniqueName: \"kubernetes.io/projected/6dfa14d1-afe4-497c-9c5e-062e742d53bf-kube-api-access-6jjgl\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.722164 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.722050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-metrics-certs\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.722805 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.722780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfa14d1-afe4-497c-9c5e-062e742d53bf-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.724602 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.724573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-metrics-certs\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.724743 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.724607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-maas-api-tls\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.730786 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.730756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jjgl\" (UniqueName: \"kubernetes.io/projected/6dfa14d1-afe4-497c-9c5e-062e742d53bf-kube-api-access-6jjgl\") pod \"maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:39.820418 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:39.820302    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:40.108633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.108503    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"]
Sep 17 14:19:40.113897 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.113868    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"]
Sep 17 14:19:40.114046 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.113994    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:19:40.114460 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:19:40.114429    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfa14d1_afe4_497c_9c5e_062e742d53bf.slice/crio-dafc16e3bbf0c14fc34ddb1f408c21db2b7795b4c8e5a898461fe05c2294cdeb WatchSource:0}: Error finding container dafc16e3bbf0c14fc34ddb1f408c21db2b7795b4c8e5a898461fe05c2294cdeb: Status 404 returned error can't find the container with id dafc16e3bbf0c14fc34ddb1f408c21db2b7795b4c8e5a898461fe05c2294cdeb
Sep 17 14:19:40.116517 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.116500    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9e4372d6-dockercfg-8q9v8\""
Sep 17 14:19:40.116821 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.116804    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9e4372d6\""
Sep 17 14:19:40.124763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.124733    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"]
Sep 17 14:19:40.135136 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.135021    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"]
Sep 17 14:19:40.139202 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.139175    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:19:40.148543 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.148510    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"]
Sep 17 14:19:40.227933 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.227894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5b9e91-2860-4bef-a793-81fc9087cb14-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv\" (UID: \"0d5b9e91-2860-4bef-a793-81fc9087cb14\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:19:40.228100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.227957    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd4kx\" (UniqueName: \"kubernetes.io/projected/0d5b9e91-2860-4bef-a793-81fc9087cb14-kube-api-access-zd4kx\") pod \"payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv\" (UID: \"0d5b9e91-2860-4bef-a793-81fc9087cb14\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:19:40.228100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.228012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc3efed-e393-493d-a85a-47b83e2497f5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5\" (UID: \"cdc3efed-e393-493d-a85a-47b83e2497f5\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:19:40.228100 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.228030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbc2t\" (UniqueName: \"kubernetes.io/projected/cdc3efed-e393-493d-a85a-47b83e2497f5-kube-api-access-kbc2t\") pod \"payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5\" (UID: \"cdc3efed-e393-493d-a85a-47b83e2497f5\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:19:40.329547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.329500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5b9e91-2860-4bef-a793-81fc9087cb14-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv\" (UID: \"0d5b9e91-2860-4bef-a793-81fc9087cb14\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:19:40.329731 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.329563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd4kx\" (UniqueName: \"kubernetes.io/projected/0d5b9e91-2860-4bef-a793-81fc9087cb14-kube-api-access-zd4kx\") pod \"payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv\" (UID: \"0d5b9e91-2860-4bef-a793-81fc9087cb14\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:19:40.329731 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.329599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc3efed-e393-493d-a85a-47b83e2497f5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5\" (UID: \"cdc3efed-e393-493d-a85a-47b83e2497f5\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:19:40.329731 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.329623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbc2t\" (UniqueName: \"kubernetes.io/projected/cdc3efed-e393-493d-a85a-47b83e2497f5-kube-api-access-kbc2t\") pod \"payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5\" (UID: \"cdc3efed-e393-493d-a85a-47b83e2497f5\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:19:40.330314 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.330289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5b9e91-2860-4bef-a793-81fc9087cb14-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv\" (UID: \"0d5b9e91-2860-4bef-a793-81fc9087cb14\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:19:40.330444 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.330406    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc3efed-e393-493d-a85a-47b83e2497f5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5\" (UID: \"cdc3efed-e393-493d-a85a-47b83e2497f5\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:19:40.338156 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.338122    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbc2t\" (UniqueName: \"kubernetes.io/projected/cdc3efed-e393-493d-a85a-47b83e2497f5-kube-api-access-kbc2t\") pod \"payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5\" (UID: \"cdc3efed-e393-493d-a85a-47b83e2497f5\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:19:40.338269 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.338136    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd4kx\" (UniqueName: \"kubernetes.io/projected/0d5b9e91-2860-4bef-a793-81fc9087cb14-kube-api-access-zd4kx\") pod \"payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv\" (UID: \"0d5b9e91-2860-4bef-a793-81fc9087cb14\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:19:40.435991 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.435892    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:19:40.452456 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.452416    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:19:40.717511 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.717478    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"]
Sep 17 14:19:40.719093 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:19:40.719062    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5b9e91_2860_4bef_a793_81fc9087cb14.slice/crio-285e60526d7c9b0d068750a4819f59bf6ade3a351c4343cad462b061e33f761b WatchSource:0}: Error finding container 285e60526d7c9b0d068750a4819f59bf6ade3a351c4343cad462b061e33f761b: Status 404 returned error can't find the container with id 285e60526d7c9b0d068750a4819f59bf6ade3a351c4343cad462b061e33f761b
Sep 17 14:19:40.741857 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.741825    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"]
Sep 17 14:19:40.743848 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:19:40.743820    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc3efed_e393_493d_a85a_47b83e2497f5.slice/crio-67d269502e164860972b60d27ff73fcb0812731a9293e4b110d232e02c2a5739 WatchSource:0}: Error finding container 67d269502e164860972b60d27ff73fcb0812731a9293e4b110d232e02c2a5739: Status 404 returned error can't find the container with id 67d269502e164860972b60d27ff73fcb0812731a9293e4b110d232e02c2a5739
Sep 17 14:19:40.751621 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.751580    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv" event={"ID":"0d5b9e91-2860-4bef-a793-81fc9087cb14","Type":"ContainerStarted","Data":"285e60526d7c9b0d068750a4819f59bf6ade3a351c4343cad462b061e33f761b"}
Sep 17 14:19:40.753170 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.753135    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq" event={"ID":"6dfa14d1-afe4-497c-9c5e-062e742d53bf","Type":"ContainerStarted","Data":"dafc16e3bbf0c14fc34ddb1f408c21db2b7795b4c8e5a898461fe05c2294cdeb"}
Sep 17 14:19:40.754510 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:40.754484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5" event={"ID":"cdc3efed-e393-493d-a85a-47b83e2497f5","Type":"ContainerStarted","Data":"67d269502e164860972b60d27ff73fcb0812731a9293e4b110d232e02c2a5739"}
Sep 17 14:19:41.762525 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:41.762468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv" event={"ID":"0d5b9e91-2860-4bef-a793-81fc9087cb14","Type":"ContainerStarted","Data":"31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167"}
Sep 17 14:19:41.762991 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:41.762556    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:19:41.774342 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:41.774299    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq" event={"ID":"6dfa14d1-afe4-497c-9c5e-062e742d53bf","Type":"ContainerStarted","Data":"1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c"}
Sep 17 14:19:41.774584 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:41.774424    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:41.776313 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:41.776276    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5" event={"ID":"cdc3efed-e393-493d-a85a-47b83e2497f5","Type":"ContainerStarted","Data":"6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26"}
Sep 17 14:19:41.776468 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:41.776383    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:19:41.788258 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:41.788183    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv" podStartSLOduration=1.7881613060000001 podStartE2EDuration="1.788161306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:41.787334538 +0000 UTC m=+879.378510301" watchObservedRunningTime="2026-09-17 14:19:41.788161306 +0000 UTC m=+879.379337069"
Sep 17 14:19:41.805382 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:41.805277    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5" podStartSLOduration=1.805252179 podStartE2EDuration="1.805252179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:41.804889711 +0000 UTC m=+879.396065549" watchObservedRunningTime="2026-09-17 14:19:41.805252179 +0000 UTC m=+879.396427942"
Sep 17 14:19:41.824187 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:41.824126    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq" podStartSLOduration=2.079558482 podStartE2EDuration="2.824106124s" totalImagesPullingTime="744.547642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:41.821429174 +0000 UTC m=+879.412604938" watchObservedRunningTime="2026-09-17 14:19:41.824106124 +0000 UTC m=+879.415281940"
Sep 17 14:19:42.321449 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:42.321408    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:42.321637 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:42.321466    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:42.334696 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:42.334661    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:42.796048 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:42.796017    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:19:47.788078 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:47.788046    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:19:52.785312 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:52.785279    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:19:52.785761 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:52.785350    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:19:56.473383 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.473254    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"]
Sep 17 14:19:56.478267 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.478238    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.481013 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.480987    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fcd9f3\"/\"openshift-service-ca.crt\""
Sep 17 14:19:56.482098 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.482070    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fcd9f3\"/\"default-dockercfg-9sc5l\""
Sep 17 14:19:56.482411 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.482385    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fcd9f3\"/\"simulator-w4-fcd9f3-kserve-self-signed-certs\""
Sep 17 14:19:56.482548 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.482228    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fcd9f3\"/\"kube-root-ca.crt\""
Sep 17 14:19:56.485954 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.485924    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"]
Sep 17 14:19:56.605687 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.605647    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-model-cache\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.605687 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.605693    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-home\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.605918 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.605774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p29qw\" (UniqueName: \"kubernetes.io/projected/d08f7a56-5408-4885-a717-c985fc5f911f-kube-api-access-p29qw\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.605918 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.605856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-tmp-dir\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.605918 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.605876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08f7a56-5408-4885-a717-c985fc5f911f-tls-certs\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.605918 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.605906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-dshm\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.706613 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.706565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-tmp-dir\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.706613 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.706609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08f7a56-5408-4885-a717-c985fc5f911f-tls-certs\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.706854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.706751    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-dshm\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.706854 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.706829    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-model-cache\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.706971 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.706866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-home\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.706971 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.706925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p29qw\" (UniqueName: \"kubernetes.io/projected/d08f7a56-5408-4885-a717-c985fc5f911f-kube-api-access-p29qw\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.707077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.707006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-tmp-dir\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.707248 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.707222    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-home\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.707350 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.707266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-model-cache\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.709072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.709050    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-dshm\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.709504 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.709483    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08f7a56-5408-4885-a717-c985fc5f911f-tls-certs\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.716205 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.716171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p29qw\" (UniqueName: \"kubernetes.io/projected/d08f7a56-5408-4885-a717-c985fc5f911f-kube-api-access-p29qw\") pod \"simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") " pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:56.805889 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:56.805844    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:19:57.102271 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:57.102243    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"]
Sep 17 14:19:57.104947 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:19:57.104924    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd08f7a56_5408_4885_a717_c985fc5f911f.slice/crio-8ad005ad7a77c7d930ad040eb47c5ad2093c005f50412e246716e5c2a9352302 WatchSource:0}: Error finding container 8ad005ad7a77c7d930ad040eb47c5ad2093c005f50412e246716e5c2a9352302: Status 404 returned error can't find the container with id 8ad005ad7a77c7d930ad040eb47c5ad2093c005f50412e246716e5c2a9352302
Sep 17 14:19:57.874164 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:57.874119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2" event={"ID":"d08f7a56-5408-4885-a717-c985fc5f911f","Type":"ContainerStarted","Data":"41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a"}
Sep 17 14:19:57.874609 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:57.874175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2" event={"ID":"d08f7a56-5408-4885-a717-c985fc5f911f","Type":"ContainerStarted","Data":"8ad005ad7a77c7d930ad040eb47c5ad2093c005f50412e246716e5c2a9352302"}
Sep 17 14:19:57.895879 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:19:57.895814    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2" podStartSLOduration=1.895792439 podStartE2EDuration="1.895792439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:57.893284404 +0000 UTC m=+895.484460169" watchObservedRunningTime="2026-09-17 14:19:57.895792439 +0000 UTC m=+895.486968254"
Sep 17 14:20:06.806868 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:06.806825    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:20:06.807410 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:06.806883    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:20:06.820115 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:06.820076    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:20:06.931826 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:06.931788    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:20:07.865341 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:07.865294    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"]
Sep 17 14:20:07.866426 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:07.865956    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq" podUID="6dfa14d1-afe4-497c-9c5e-062e742d53bf" containerName="maas-api" containerID="cri-o://1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c" gracePeriod=30
Sep 17 14:20:07.921397 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:07.920282    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"]
Sep 17 14:20:07.921397 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:07.920575    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5" podUID="cdc3efed-e393-493d-a85a-47b83e2497f5" containerName="payload-processing" containerID="cri-o://6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26" gracePeriod=30
Sep 17 14:20:07.923588 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:07.923556    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"]
Sep 17 14:20:07.923812 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:07.923784    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv" podUID="0d5b9e91-2860-4bef-a793-81fc9087cb14" containerName="payload-pre-processing" containerID="cri-o://31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167" gracePeriod=30
Sep 17 14:20:08.204633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.204602    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:20:08.211676 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.211640    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-maas-api-tls\") pod \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") "
Sep 17 14:20:08.211865 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.211694    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfa14d1-afe4-497c-9c5e-062e742d53bf-openshift-service-ca\") pod \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") "
Sep 17 14:20:08.211865 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.211727    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-metrics-certs\") pod \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") "
Sep 17 14:20:08.211865 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.211830    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jjgl\" (UniqueName: \"kubernetes.io/projected/6dfa14d1-afe4-497c-9c5e-062e742d53bf-kube-api-access-6jjgl\") pod \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\" (UID: \"6dfa14d1-afe4-497c-9c5e-062e742d53bf\") "
Sep 17 14:20:08.212383 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.212326    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfa14d1-afe4-497c-9c5e-062e742d53bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6dfa14d1-afe4-497c-9c5e-062e742d53bf" (UID: "6dfa14d1-afe4-497c-9c5e-062e742d53bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:20:08.217669 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.217621    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfa14d1-afe4-497c-9c5e-062e742d53bf-kube-api-access-6jjgl" (OuterVolumeSpecName: "kube-api-access-6jjgl") pod "6dfa14d1-afe4-497c-9c5e-062e742d53bf" (UID: "6dfa14d1-afe4-497c-9c5e-062e742d53bf"). InnerVolumeSpecName "kube-api-access-6jjgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:20:08.219648 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.219595    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6dfa14d1-afe4-497c-9c5e-062e742d53bf" (UID: "6dfa14d1-afe4-497c-9c5e-062e742d53bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:20:08.221037 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.220995    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6dfa14d1-afe4-497c-9c5e-062e742d53bf" (UID: "6dfa14d1-afe4-497c-9c5e-062e742d53bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:20:08.253012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.252978    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:20:08.256343 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.256314    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:20:08.312538 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.312496    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc3efed-e393-493d-a85a-47b83e2497f5-plugins-config-volume\") pod \"cdc3efed-e393-493d-a85a-47b83e2497f5\" (UID: \"cdc3efed-e393-493d-a85a-47b83e2497f5\") "
Sep 17 14:20:08.312748 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.312553    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd4kx\" (UniqueName: \"kubernetes.io/projected/0d5b9e91-2860-4bef-a793-81fc9087cb14-kube-api-access-zd4kx\") pod \"0d5b9e91-2860-4bef-a793-81fc9087cb14\" (UID: \"0d5b9e91-2860-4bef-a793-81fc9087cb14\") "
Sep 17 14:20:08.312748 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.312601    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbc2t\" (UniqueName: \"kubernetes.io/projected/cdc3efed-e393-493d-a85a-47b83e2497f5-kube-api-access-kbc2t\") pod \"cdc3efed-e393-493d-a85a-47b83e2497f5\" (UID: \"cdc3efed-e393-493d-a85a-47b83e2497f5\") "
Sep 17 14:20:08.312748 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.312644    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5b9e91-2860-4bef-a793-81fc9087cb14-plugins-config-volume\") pod \"0d5b9e91-2860-4bef-a793-81fc9087cb14\" (UID: \"0d5b9e91-2860-4bef-a793-81fc9087cb14\") "
Sep 17 14:20:08.312982 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.312962    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:08.313038 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.312990    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfa14d1-afe4-497c-9c5e-062e742d53bf-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:08.313038 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.312991    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc3efed-e393-493d-a85a-47b83e2497f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cdc3efed-e393-493d-a85a-47b83e2497f5" (UID: "cdc3efed-e393-493d-a85a-47b83e2497f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:20:08.313038 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.313002    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfa14d1-afe4-497c-9c5e-062e742d53bf-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:08.313181 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.313057    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jjgl\" (UniqueName: \"kubernetes.io/projected/6dfa14d1-afe4-497c-9c5e-062e742d53bf-kube-api-access-6jjgl\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:08.313181 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.313086    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5b9e91-2860-4bef-a793-81fc9087cb14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d5b9e91-2860-4bef-a793-81fc9087cb14" (UID: "0d5b9e91-2860-4bef-a793-81fc9087cb14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:20:08.314891 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.314859    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5b9e91-2860-4bef-a793-81fc9087cb14-kube-api-access-zd4kx" (OuterVolumeSpecName: "kube-api-access-zd4kx") pod "0d5b9e91-2860-4bef-a793-81fc9087cb14" (UID: "0d5b9e91-2860-4bef-a793-81fc9087cb14"). InnerVolumeSpecName "kube-api-access-zd4kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:20:08.315042 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.314938    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc3efed-e393-493d-a85a-47b83e2497f5-kube-api-access-kbc2t" (OuterVolumeSpecName: "kube-api-access-kbc2t") pod "cdc3efed-e393-493d-a85a-47b83e2497f5" (UID: "cdc3efed-e393-493d-a85a-47b83e2497f5"). InnerVolumeSpecName "kube-api-access-kbc2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:20:08.413756 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.413709    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc3efed-e393-493d-a85a-47b83e2497f5-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:08.413756 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.413746    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd4kx\" (UniqueName: \"kubernetes.io/projected/0d5b9e91-2860-4bef-a793-81fc9087cb14-kube-api-access-zd4kx\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:08.413756 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.413755    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbc2t\" (UniqueName: \"kubernetes.io/projected/cdc3efed-e393-493d-a85a-47b83e2497f5-kube-api-access-kbc2t\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:08.413756 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.413766    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5b9e91-2860-4bef-a793-81fc9087cb14-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:08.934446 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.934410    2560 generic.go:356] "Generic (PLEG): container finished" podID="0d5b9e91-2860-4bef-a793-81fc9087cb14" containerID="31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167" exitCode=0
Sep 17 14:20:08.934943 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.934454    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"
Sep 17 14:20:08.934943 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.934510    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv" event={"ID":"0d5b9e91-2860-4bef-a793-81fc9087cb14","Type":"ContainerDied","Data":"31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167"}
Sep 17 14:20:08.934943 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.934565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv" event={"ID":"0d5b9e91-2860-4bef-a793-81fc9087cb14","Type":"ContainerDied","Data":"285e60526d7c9b0d068750a4819f59bf6ade3a351c4343cad462b061e33f761b"}
Sep 17 14:20:08.935307 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.934594    2560 scope.go:117] "RemoveContainer" containerID="31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167"
Sep 17 14:20:08.937498 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.936864    2560 generic.go:356] "Generic (PLEG): container finished" podID="6dfa14d1-afe4-497c-9c5e-062e742d53bf" containerID="1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c" exitCode=0
Sep 17 14:20:08.937498 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.936916    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"
Sep 17 14:20:08.937498 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.936988    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq" event={"ID":"6dfa14d1-afe4-497c-9c5e-062e742d53bf","Type":"ContainerDied","Data":"1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c"}
Sep 17 14:20:08.937498 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.937040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq" event={"ID":"6dfa14d1-afe4-497c-9c5e-062e742d53bf","Type":"ContainerDied","Data":"dafc16e3bbf0c14fc34ddb1f408c21db2b7795b4c8e5a898461fe05c2294cdeb"}
Sep 17 14:20:08.939183 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.939153    2560 generic.go:356] "Generic (PLEG): container finished" podID="cdc3efed-e393-493d-a85a-47b83e2497f5" containerID="6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26" exitCode=0
Sep 17 14:20:08.939343 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.939210    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"
Sep 17 14:20:08.939343 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.939230    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5" event={"ID":"cdc3efed-e393-493d-a85a-47b83e2497f5","Type":"ContainerDied","Data":"6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26"}
Sep 17 14:20:08.939343 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.939275    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5" event={"ID":"cdc3efed-e393-493d-a85a-47b83e2497f5","Type":"ContainerDied","Data":"67d269502e164860972b60d27ff73fcb0812731a9293e4b110d232e02c2a5739"}
Sep 17 14:20:08.959484 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.959223    2560 scope.go:117] "RemoveContainer" containerID="31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167"
Sep 17 14:20:08.960020 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:20:08.959960    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167\": container with ID starting with 31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167 not found: ID does not exist" containerID="31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167"
Sep 17 14:20:08.960409 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.960006    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167"} err="failed to get container status \"31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167\": rpc error: code = NotFound desc = could not find container \"31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167\": container with ID starting with 31c72602126b1a321fb3a9dee7a8eefa677300f9251c6f55d99af3e5ed2c1167 not found: ID does not exist"
Sep 17 14:20:08.960409 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.960244    2560 scope.go:117] "RemoveContainer" containerID="1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c"
Sep 17 14:20:08.973295 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.973270    2560 scope.go:117] "RemoveContainer" containerID="1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c"
Sep 17 14:20:08.973735 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:20:08.973679    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c\": container with ID starting with 1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c not found: ID does not exist" containerID="1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c"
Sep 17 14:20:08.973901 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.973730    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c"} err="failed to get container status \"1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c\": rpc error: code = NotFound desc = could not find container \"1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c\": container with ID starting with 1b6e1cf2091cbe7939d6ed44a7f45d883e8effeca05bc5bc31b391cbe485e58c not found: ID does not exist"
Sep 17 14:20:08.973901 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.973754    2560 scope.go:117] "RemoveContainer" containerID="6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26"
Sep 17 14:20:08.985203 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.985177    2560 scope.go:117] "RemoveContainer" containerID="6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26"
Sep 17 14:20:08.985612 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:20:08.985588    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26\": container with ID starting with 6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26 not found: ID does not exist" containerID="6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26"
Sep 17 14:20:08.985691 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.985621    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26"} err="failed to get container status \"6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26\": rpc error: code = NotFound desc = could not find container \"6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26\": container with ID starting with 6c6ab68719ead1dc3d8efa2bf9592047910ea476a8ee4c9cc239f3e1fa374f26 not found: ID does not exist"
Sep 17 14:20:08.989074 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.989042    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"]
Sep 17 14:20:08.997050 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:08.997010    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e4372d6-59bf47c58c-46dlq"]
Sep 17 14:20:09.009413 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:09.009343    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"]
Sep 17 14:20:09.015665 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:09.015628    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e4372d6-87658f6db-bz9pv"]
Sep 17 14:20:09.025951 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:09.025908    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"]
Sep 17 14:20:09.030032 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:09.029995    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e4372d6-649d68cd9d-rq6m5"]
Sep 17 14:20:10.967876 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:10.967832    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d5b9e91-2860-4bef-a793-81fc9087cb14" path="/var/lib/kubelet/pods/0d5b9e91-2860-4bef-a793-81fc9087cb14/volumes"
Sep 17 14:20:10.968492 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:10.968471    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dfa14d1-afe4-497c-9c5e-062e742d53bf" path="/var/lib/kubelet/pods/6dfa14d1-afe4-497c-9c5e-062e742d53bf/volumes"
Sep 17 14:20:10.969012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:10.968988    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdc3efed-e393-493d-a85a-47b83e2497f5" path="/var/lib/kubelet/pods/cdc3efed-e393-493d-a85a-47b83e2497f5/volumes"
Sep 17 14:20:13.558747 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.558699    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"]
Sep 17 14:20:13.559279 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.559248    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5b9e91-2860-4bef-a793-81fc9087cb14" containerName="payload-pre-processing"
Sep 17 14:20:13.559279 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.559268    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5b9e91-2860-4bef-a793-81fc9087cb14" containerName="payload-pre-processing"
Sep 17 14:20:13.559482 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.559280    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dfa14d1-afe4-497c-9c5e-062e742d53bf" containerName="maas-api"
Sep 17 14:20:13.559482 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.559291    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dfa14d1-afe4-497c-9c5e-062e742d53bf" containerName="maas-api"
Sep 17 14:20:13.559482 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.559316    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc3efed-e393-493d-a85a-47b83e2497f5" containerName="payload-processing"
Sep 17 14:20:13.559482 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.559324    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc3efed-e393-493d-a85a-47b83e2497f5" containerName="payload-processing"
Sep 17 14:20:13.559706 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.559609    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dfa14d1-afe4-497c-9c5e-062e742d53bf" containerName="maas-api"
Sep 17 14:20:13.559706 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.559630    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5b9e91-2860-4bef-a793-81fc9087cb14" containerName="payload-pre-processing"
Sep 17 14:20:13.559706 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.559646    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdc3efed-e393-493d-a85a-47b83e2497f5" containerName="payload-processing"
Sep 17 14:20:13.564812 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.564777    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.567345 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.567316    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-60f59148-serving-cert\""
Sep 17 14:20:13.574824 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.574799    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"]
Sep 17 14:20:13.662434 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.662389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-maas-api-tls\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.662618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.662517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9qr8\" (UniqueName: \"kubernetes.io/projected/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-kube-api-access-r9qr8\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.662618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.662545    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-openshift-service-ca\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.662618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.662562    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-metrics-certs\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.763878 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.763831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-maas-api-tls\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.764069 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.763943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qr8\" (UniqueName: \"kubernetes.io/projected/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-kube-api-access-r9qr8\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.764069 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.763966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-openshift-service-ca\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.764069 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.763982    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-metrics-certs\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.764069 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:20:13.764035    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-serving-cert: secret "maas-api-e2e-ait-60f59148-serving-cert" not found
Sep 17 14:20:13.764201 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:20:13.764128    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-maas-api-tls podName:0d3c8b8f-2151-4a69-9ccd-39d365463f8a nodeName:}" failed. No retries permitted until 2026-09-17 14:20:14.264109305 +0000 UTC m=+911.855285063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-maas-api-tls") pod "maas-api-e2e-ait-60f59148-546fcb977-xfjtt" (UID: "0d3c8b8f-2151-4a69-9ccd-39d365463f8a") : secret "maas-api-e2e-ait-60f59148-serving-cert" not found
Sep 17 14:20:13.764650 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.764623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-openshift-service-ca\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.766606 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.766586    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-metrics-certs\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:13.773475 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:13.773443    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qr8\" (UniqueName: \"kubernetes.io/projected/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-kube-api-access-r9qr8\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:14.179155 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.179113    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"]
Sep 17 14:20:14.182908 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.182882    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:14.185774 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.185745    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-60f59148\""
Sep 17 14:20:14.185911 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.185835    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-60f59148-dockercfg-lb9zz\""
Sep 17 14:20:14.195608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.195574    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"]
Sep 17 14:20:14.230082 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.230048    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"]
Sep 17 14:20:14.233932 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.233909    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:14.243027 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.242994    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"]
Sep 17 14:20:14.269614 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.269573    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxpmv\" (UniqueName: \"kubernetes.io/projected/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-kube-api-access-sxpmv\") pod \"payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh\" (UID: \"f5905c8b-8c5e-4e06-9521-82c78e72e3ff\") " pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:14.269786 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.269634    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-maas-api-tls\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:14.269786 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.269730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smztp\" (UniqueName: \"kubernetes.io/projected/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-kube-api-access-smztp\") pod \"payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl\" (UID: \"96a0077d-cbb4-4a1a-af48-025f3f9e2baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:14.269786 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.269779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl\" (UID: \"96a0077d-cbb4-4a1a-af48-025f3f9e2baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:14.269932 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.269798    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh\" (UID: \"f5905c8b-8c5e-4e06-9521-82c78e72e3ff\") " pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:14.272068 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.272046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-maas-api-tls\") pod \"maas-api-e2e-ait-60f59148-546fcb977-xfjtt\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:14.371513 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.371466    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxpmv\" (UniqueName: \"kubernetes.io/projected/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-kube-api-access-sxpmv\") pod \"payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh\" (UID: \"f5905c8b-8c5e-4e06-9521-82c78e72e3ff\") " pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:14.371710 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.371626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smztp\" (UniqueName: \"kubernetes.io/projected/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-kube-api-access-smztp\") pod \"payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl\" (UID: \"96a0077d-cbb4-4a1a-af48-025f3f9e2baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:14.371710 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.371682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl\" (UID: \"96a0077d-cbb4-4a1a-af48-025f3f9e2baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:14.371710 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.371706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh\" (UID: \"f5905c8b-8c5e-4e06-9521-82c78e72e3ff\") " pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:14.372421 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.372399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl\" (UID: \"96a0077d-cbb4-4a1a-af48-025f3f9e2baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:14.372673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.372651    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh\" (UID: \"f5905c8b-8c5e-4e06-9521-82c78e72e3ff\") " pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:14.381807 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.381772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxpmv\" (UniqueName: \"kubernetes.io/projected/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-kube-api-access-sxpmv\") pod \"payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh\" (UID: \"f5905c8b-8c5e-4e06-9521-82c78e72e3ff\") " pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:14.381807 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.381794    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smztp\" (UniqueName: \"kubernetes.io/projected/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-kube-api-access-smztp\") pod \"payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl\" (UID: \"96a0077d-cbb4-4a1a-af48-025f3f9e2baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:14.478475 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.478383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:14.494794 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.494758    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:14.546235 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.545947    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:14.758993 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.758958    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"]
Sep 17 14:20:14.761599 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:20:14.761568    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3c8b8f_2151_4a69_9ccd_39d365463f8a.slice/crio-20f3e79aa6fcd01e00cf87426e72bcbf883aa1cb14dd708a8ebcd300b5b61e7c WatchSource:0}: Error finding container 20f3e79aa6fcd01e00cf87426e72bcbf883aa1cb14dd708a8ebcd300b5b61e7c: Status 404 returned error can't find the container with id 20f3e79aa6fcd01e00cf87426e72bcbf883aa1cb14dd708a8ebcd300b5b61e7c
Sep 17 14:20:14.778618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.778589    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"]
Sep 17 14:20:14.781088 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:20:14.781054    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5905c8b_8c5e_4e06_9521_82c78e72e3ff.slice/crio-318c32f7b545d15849d02b29c608612750c120a81a9407c5daf8d235d3237684 WatchSource:0}: Error finding container 318c32f7b545d15849d02b29c608612750c120a81a9407c5daf8d235d3237684: Status 404 returned error can't find the container with id 318c32f7b545d15849d02b29c608612750c120a81a9407c5daf8d235d3237684
Sep 17 14:20:14.828803 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.828774    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"]
Sep 17 14:20:14.831697 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:20:14.831663    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a0077d_cbb4_4a1a_af48_025f3f9e2baf.slice/crio-10775fd8db0efaddc47591d849528748425e28d48de20bcaaa02d65e7f7a91c7 WatchSource:0}: Error finding container 10775fd8db0efaddc47591d849528748425e28d48de20bcaaa02d65e7f7a91c7: Status 404 returned error can't find the container with id 10775fd8db0efaddc47591d849528748425e28d48de20bcaaa02d65e7f7a91c7
Sep 17 14:20:14.976687 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.976644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt" event={"ID":"0d3c8b8f-2151-4a69-9ccd-39d365463f8a","Type":"ContainerStarted","Data":"20f3e79aa6fcd01e00cf87426e72bcbf883aa1cb14dd708a8ebcd300b5b61e7c"}
Sep 17 14:20:14.978617 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.978565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl" event={"ID":"96a0077d-cbb4-4a1a-af48-025f3f9e2baf","Type":"ContainerStarted","Data":"1424e8cca5313f7359534ab3f068fc7ac34b456ca44c306ec106411c83accaf6"}
Sep 17 14:20:14.978811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.978631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl" event={"ID":"96a0077d-cbb4-4a1a-af48-025f3f9e2baf","Type":"ContainerStarted","Data":"10775fd8db0efaddc47591d849528748425e28d48de20bcaaa02d65e7f7a91c7"}
Sep 17 14:20:14.978811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.978662    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:14.980497 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.980465    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh" event={"ID":"f5905c8b-8c5e-4e06-9521-82c78e72e3ff","Type":"ContainerStarted","Data":"edb66621e5086a94d58de16d1dfef31ec70d0aca30ec02f39a9a6c14aff40534"}
Sep 17 14:20:14.980655 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.980513    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh" event={"ID":"f5905c8b-8c5e-4e06-9521-82c78e72e3ff","Type":"ContainerStarted","Data":"318c32f7b545d15849d02b29c608612750c120a81a9407c5daf8d235d3237684"}
Sep 17 14:20:14.980655 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.980577    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:14.999010 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:14.998953    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl" podStartSLOduration=0.998936187 podStartE2EDuration="998.936187ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:20:14.997420958 +0000 UTC m=+912.588596720" watchObservedRunningTime="2026-09-17 14:20:14.998936187 +0000 UTC m=+912.590111949"
Sep 17 14:20:15.017409 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:15.017259    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh" podStartSLOduration=1.017241132 podStartE2EDuration="1.017241132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:20:15.015121896 +0000 UTC m=+912.606297659" watchObservedRunningTime="2026-09-17 14:20:15.017241132 +0000 UTC m=+912.608416893"
Sep 17 14:20:15.988054 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:15.988011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt" event={"ID":"0d3c8b8f-2151-4a69-9ccd-39d365463f8a","Type":"ContainerStarted","Data":"73884b1493aa5ce93fe62872100401f4ec84d7cc382afb6fb5f0b987c5f121e9"}
Sep 17 14:20:16.015093 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:16.015029    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt" podStartSLOduration=2.571323025 podStartE2EDuration="3.015010192s" totalImagesPullingTime="443.687167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:20:16.013463286 +0000 UTC m=+913.604639049" watchObservedRunningTime="2026-09-17 14:20:16.015010192 +0000 UTC m=+913.606185954"
Sep 17 14:20:16.995430 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:16.995392    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:23.006071 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:23.006036    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:25.990441 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:25.990406    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:25.990842 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:25.990481    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:42.908496 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:42.908458    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"]
Sep 17 14:20:42.909027 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:42.908696    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt" podUID="0d3c8b8f-2151-4a69-9ccd-39d365463f8a" containerName="maas-api" containerID="cri-o://73884b1493aa5ce93fe62872100401f4ec84d7cc382afb6fb5f0b987c5f121e9" gracePeriod=30
Sep 17 14:20:42.956160 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:42.956114    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"]
Sep 17 14:20:42.956426 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:42.956397    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh" podUID="f5905c8b-8c5e-4e06-9521-82c78e72e3ff" containerName="payload-processing" containerID="cri-o://edb66621e5086a94d58de16d1dfef31ec70d0aca30ec02f39a9a6c14aff40534" gracePeriod=30
Sep 17 14:20:42.968643 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:42.968603    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"]
Sep 17 14:20:42.968959 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:42.968901    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl" podUID="96a0077d-cbb4-4a1a-af48-025f3f9e2baf" containerName="payload-pre-processing" containerID="cri-o://1424e8cca5313f7359534ab3f068fc7ac34b456ca44c306ec106411c83accaf6" gracePeriod=30
Sep 17 14:20:43.001181 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.001128    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt" podUID="0d3c8b8f-2151-4a69-9ccd-39d365463f8a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.48:8443/health\": dial tcp 10.133.0.48:8443: connect: connection refused"
Sep 17 14:20:43.154340 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.154304    2560 generic.go:356] "Generic (PLEG): container finished" podID="96a0077d-cbb4-4a1a-af48-025f3f9e2baf" containerID="1424e8cca5313f7359534ab3f068fc7ac34b456ca44c306ec106411c83accaf6" exitCode=0
Sep 17 14:20:43.154470 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.154421    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl" event={"ID":"96a0077d-cbb4-4a1a-af48-025f3f9e2baf","Type":"ContainerDied","Data":"1424e8cca5313f7359534ab3f068fc7ac34b456ca44c306ec106411c83accaf6"}
Sep 17 14:20:43.158153 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.156805    2560 generic.go:356] "Generic (PLEG): container finished" podID="f5905c8b-8c5e-4e06-9521-82c78e72e3ff" containerID="edb66621e5086a94d58de16d1dfef31ec70d0aca30ec02f39a9a6c14aff40534" exitCode=0
Sep 17 14:20:43.158153 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.157219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh" event={"ID":"f5905c8b-8c5e-4e06-9521-82c78e72e3ff","Type":"ContainerDied","Data":"edb66621e5086a94d58de16d1dfef31ec70d0aca30ec02f39a9a6c14aff40534"}
Sep 17 14:20:43.159519 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.159320    2560 generic.go:356] "Generic (PLEG): container finished" podID="0d3c8b8f-2151-4a69-9ccd-39d365463f8a" containerID="73884b1493aa5ce93fe62872100401f4ec84d7cc382afb6fb5f0b987c5f121e9" exitCode=0
Sep 17 14:20:43.159519 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.159383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt" event={"ID":"0d3c8b8f-2151-4a69-9ccd-39d365463f8a","Type":"ContainerDied","Data":"73884b1493aa5ce93fe62872100401f4ec84d7cc382afb6fb5f0b987c5f121e9"}
Sep 17 14:20:43.256987 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.256941    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:43.291676 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.291635    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:43.308429 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.308399    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:43.376380 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.376325    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9qr8\" (UniqueName: \"kubernetes.io/projected/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-kube-api-access-r9qr8\") pod \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") "
Sep 17 14:20:43.376582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.376433    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-maas-api-tls\") pod \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") "
Sep 17 14:20:43.376582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.376471    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-metrics-certs\") pod \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") "
Sep 17 14:20:43.376582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.376508    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-openshift-service-ca\") pod \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\" (UID: \"0d3c8b8f-2151-4a69-9ccd-39d365463f8a\") "
Sep 17 14:20:43.376899 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.376858    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d3c8b8f-2151-4a69-9ccd-39d365463f8a" (UID: "0d3c8b8f-2151-4a69-9ccd-39d365463f8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:20:43.378593 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.378564    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d3c8b8f-2151-4a69-9ccd-39d365463f8a" (UID: "0d3c8b8f-2151-4a69-9ccd-39d365463f8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:20:43.378699 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.378621    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-kube-api-access-r9qr8" (OuterVolumeSpecName: "kube-api-access-r9qr8") pod "0d3c8b8f-2151-4a69-9ccd-39d365463f8a" (UID: "0d3c8b8f-2151-4a69-9ccd-39d365463f8a"). InnerVolumeSpecName "kube-api-access-r9qr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:20:43.378699 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.378677    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0d3c8b8f-2151-4a69-9ccd-39d365463f8a" (UID: "0d3c8b8f-2151-4a69-9ccd-39d365463f8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:20:43.477261 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.477150    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smztp\" (UniqueName: \"kubernetes.io/projected/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-kube-api-access-smztp\") pod \"96a0077d-cbb4-4a1a-af48-025f3f9e2baf\" (UID: \"96a0077d-cbb4-4a1a-af48-025f3f9e2baf\") "
Sep 17 14:20:43.477467 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.477295    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-plugins-config-volume\") pod \"f5905c8b-8c5e-4e06-9521-82c78e72e3ff\" (UID: \"f5905c8b-8c5e-4e06-9521-82c78e72e3ff\") "
Sep 17 14:20:43.477467 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.477371    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxpmv\" (UniqueName: \"kubernetes.io/projected/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-kube-api-access-sxpmv\") pod \"f5905c8b-8c5e-4e06-9521-82c78e72e3ff\" (UID: \"f5905c8b-8c5e-4e06-9521-82c78e72e3ff\") "
Sep 17 14:20:43.477467 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.477416    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-plugins-config-volume\") pod \"96a0077d-cbb4-4a1a-af48-025f3f9e2baf\" (UID: \"96a0077d-cbb4-4a1a-af48-025f3f9e2baf\") "
Sep 17 14:20:43.477751 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.477733    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:43.477813 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.477756    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9qr8\" (UniqueName: \"kubernetes.io/projected/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-kube-api-access-r9qr8\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:43.477813 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.477699    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5905c8b-8c5e-4e06-9521-82c78e72e3ff" (UID: "f5905c8b-8c5e-4e06-9521-82c78e72e3ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:20:43.477813 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.477771    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:43.477927 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.477825    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3c8b8f-2151-4a69-9ccd-39d365463f8a-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:43.477927 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.477882    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96a0077d-cbb4-4a1a-af48-025f3f9e2baf" (UID: "96a0077d-cbb4-4a1a-af48-025f3f9e2baf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:20:43.479597 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.479568    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-kube-api-access-sxpmv" (OuterVolumeSpecName: "kube-api-access-sxpmv") pod "f5905c8b-8c5e-4e06-9521-82c78e72e3ff" (UID: "f5905c8b-8c5e-4e06-9521-82c78e72e3ff"). InnerVolumeSpecName "kube-api-access-sxpmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:20:43.479727 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.479605    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-kube-api-access-smztp" (OuterVolumeSpecName: "kube-api-access-smztp") pod "96a0077d-cbb4-4a1a-af48-025f3f9e2baf" (UID: "96a0077d-cbb4-4a1a-af48-025f3f9e2baf"). InnerVolumeSpecName "kube-api-access-smztp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:20:43.578768 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.578720    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxpmv\" (UniqueName: \"kubernetes.io/projected/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-kube-api-access-sxpmv\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:43.578768 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.578762    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:43.578768 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.578775    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smztp\" (UniqueName: \"kubernetes.io/projected/96a0077d-cbb4-4a1a-af48-025f3f9e2baf-kube-api-access-smztp\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:43.579058 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:43.578783    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5905c8b-8c5e-4e06-9521-82c78e72e3ff-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:20:44.166959 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.166923    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"
Sep 17 14:20:44.166959 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.166941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt" event={"ID":"0d3c8b8f-2151-4a69-9ccd-39d365463f8a","Type":"ContainerDied","Data":"20f3e79aa6fcd01e00cf87426e72bcbf883aa1cb14dd708a8ebcd300b5b61e7c"}
Sep 17 14:20:44.167533 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.167014    2560 scope.go:117] "RemoveContainer" containerID="73884b1493aa5ce93fe62872100401f4ec84d7cc382afb6fb5f0b987c5f121e9"
Sep 17 14:20:44.168614 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.168533    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"
Sep 17 14:20:44.168702 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.168536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl" event={"ID":"96a0077d-cbb4-4a1a-af48-025f3f9e2baf","Type":"ContainerDied","Data":"10775fd8db0efaddc47591d849528748425e28d48de20bcaaa02d65e7f7a91c7"}
Sep 17 14:20:44.170194 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.170169    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"
Sep 17 14:20:44.170302 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.170188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh" event={"ID":"f5905c8b-8c5e-4e06-9521-82c78e72e3ff","Type":"ContainerDied","Data":"318c32f7b545d15849d02b29c608612750c120a81a9407c5daf8d235d3237684"}
Sep 17 14:20:44.178677 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.178652    2560 scope.go:117] "RemoveContainer" containerID="1424e8cca5313f7359534ab3f068fc7ac34b456ca44c306ec106411c83accaf6"
Sep 17 14:20:44.188499 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.188478    2560 scope.go:117] "RemoveContainer" containerID="edb66621e5086a94d58de16d1dfef31ec70d0aca30ec02f39a9a6c14aff40534"
Sep 17 14:20:44.192801 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.192770    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"]
Sep 17 14:20:44.197629 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.197600    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60f59148-546fcb977-xfjtt"]
Sep 17 14:20:44.207984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.207946    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"]
Sep 17 14:20:44.212405 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.212373    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60f59148-858dcbcd9-vnwdh"]
Sep 17 14:20:44.223684 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.223651    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"]
Sep 17 14:20:44.229617 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.229583    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60f59148-65bc8d87c8-zsqgl"]
Sep 17 14:20:44.966530 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.966494    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3c8b8f-2151-4a69-9ccd-39d365463f8a" path="/var/lib/kubelet/pods/0d3c8b8f-2151-4a69-9ccd-39d365463f8a/volumes"
Sep 17 14:20:44.966937 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.966922    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a0077d-cbb4-4a1a-af48-025f3f9e2baf" path="/var/lib/kubelet/pods/96a0077d-cbb4-4a1a-af48-025f3f9e2baf/volumes"
Sep 17 14:20:44.967254 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:44.967242    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5905c8b-8c5e-4e06-9521-82c78e72e3ff" path="/var/lib/kubelet/pods/f5905c8b-8c5e-4e06-9521-82c78e72e3ff/volumes"
Sep 17 14:20:52.104449 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.104398    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"]
Sep 17 14:20:52.105040 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.105021    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a0077d-cbb4-4a1a-af48-025f3f9e2baf" containerName="payload-pre-processing"
Sep 17 14:20:52.105086 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.105044    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a0077d-cbb4-4a1a-af48-025f3f9e2baf" containerName="payload-pre-processing"
Sep 17 14:20:52.105125 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.105085    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5905c8b-8c5e-4e06-9521-82c78e72e3ff" containerName="payload-processing"
Sep 17 14:20:52.105125 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.105096    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5905c8b-8c5e-4e06-9521-82c78e72e3ff" containerName="payload-processing"
Sep 17 14:20:52.105125 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.105112    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3c8b8f-2151-4a69-9ccd-39d365463f8a" containerName="maas-api"
Sep 17 14:20:52.105125 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.105122    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3c8b8f-2151-4a69-9ccd-39d365463f8a" containerName="maas-api"
Sep 17 14:20:52.105246 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.105206    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3c8b8f-2151-4a69-9ccd-39d365463f8a" containerName="maas-api"
Sep 17 14:20:52.105246 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.105220    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a0077d-cbb4-4a1a-af48-025f3f9e2baf" containerName="payload-pre-processing"
Sep 17 14:20:52.105246 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.105230    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5905c8b-8c5e-4e06-9521-82c78e72e3ff" containerName="payload-processing"
Sep 17 14:20:52.110050 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.110019    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.112705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.112672    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-99500ae5-serving-cert\""
Sep 17 14:20:52.121009 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.120970    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"]
Sep 17 14:20:52.161375 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.161323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-maas-api-tls\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.161564 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.161479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-metrics-certs\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.161608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.161563    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qpbs\" (UniqueName: \"kubernetes.io/projected/b04bde03-1a72-46d0-abd9-9ef395eda24d-kube-api-access-9qpbs\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.161608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.161587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04bde03-1a72-46d0-abd9-9ef395eda24d-openshift-service-ca\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.262090 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.262049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qpbs\" (UniqueName: \"kubernetes.io/projected/b04bde03-1a72-46d0-abd9-9ef395eda24d-kube-api-access-9qpbs\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.262090 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.262093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04bde03-1a72-46d0-abd9-9ef395eda24d-openshift-service-ca\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.262401 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.262149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-maas-api-tls\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.262401 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.262200    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-metrics-certs\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.262401 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:20:52.262339    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-serving-cert: secret "maas-api-e2e-derive-99500ae5-serving-cert" not found
Sep 17 14:20:52.262575 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:20:52.262466    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-maas-api-tls podName:b04bde03-1a72-46d0-abd9-9ef395eda24d nodeName:}" failed. No retries permitted until 2026-09-17 14:20:52.762443739 +0000 UTC m=+950.353619481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-maas-api-tls") pod "maas-api-e2e-derive-99500ae5-6678978f75-plr8m" (UID: "b04bde03-1a72-46d0-abd9-9ef395eda24d") : secret "maas-api-e2e-derive-99500ae5-serving-cert" not found
Sep 17 14:20:52.262905 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.262886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04bde03-1a72-46d0-abd9-9ef395eda24d-openshift-service-ca\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.264732 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.264710    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-metrics-certs\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.273705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.273675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qpbs\" (UniqueName: \"kubernetes.io/projected/b04bde03-1a72-46d0-abd9-9ef395eda24d-kube-api-access-9qpbs\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.767488 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.767440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-maas-api-tls\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.769991 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.769953    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-maas-api-tls\") pod \"maas-api-e2e-derive-99500ae5-6678978f75-plr8m\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:52.793037 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.792991    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"]
Sep 17 14:20:52.797088 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.797055    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:20:52.799799 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.799768    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-99500ae5\""
Sep 17 14:20:52.807287 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.807256    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"]
Sep 17 14:20:52.844621 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.844580    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"]
Sep 17 14:20:52.848107 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.848088    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:20:52.850852 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.850830    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-99500ae5-dockercfg-5d9mt\""
Sep 17 14:20:52.859539 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.859509    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"]
Sep 17 14:20:52.868747 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.868712    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4341db7b-e0b3-409f-9709-b30165676944-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n\" (UID: \"4341db7b-e0b3-409f-9709-b30165676944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:20:52.868955 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.868769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb7zg\" (UniqueName: \"kubernetes.io/projected/9f850804-bb2e-4621-8b75-74cd586e59fa-kube-api-access-zb7zg\") pod \"payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8\" (UID: \"9f850804-bb2e-4621-8b75-74cd586e59fa\") " pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:20:52.868955 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.868881    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8j4p\" (UniqueName: \"kubernetes.io/projected/4341db7b-e0b3-409f-9709-b30165676944-kube-api-access-n8j4p\") pod \"payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n\" (UID: \"4341db7b-e0b3-409f-9709-b30165676944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:20:52.868955 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.868944    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f850804-bb2e-4621-8b75-74cd586e59fa-plugins-config-volume\") pod \"payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8\" (UID: \"9f850804-bb2e-4621-8b75-74cd586e59fa\") " pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:20:52.970226 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.970190    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8j4p\" (UniqueName: \"kubernetes.io/projected/4341db7b-e0b3-409f-9709-b30165676944-kube-api-access-n8j4p\") pod \"payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n\" (UID: \"4341db7b-e0b3-409f-9709-b30165676944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:20:52.970455 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.970247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f850804-bb2e-4621-8b75-74cd586e59fa-plugins-config-volume\") pod \"payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8\" (UID: \"9f850804-bb2e-4621-8b75-74cd586e59fa\") " pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:20:52.970455 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.970308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4341db7b-e0b3-409f-9709-b30165676944-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n\" (UID: \"4341db7b-e0b3-409f-9709-b30165676944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:20:52.970455 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.970352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb7zg\" (UniqueName: \"kubernetes.io/projected/9f850804-bb2e-4621-8b75-74cd586e59fa-kube-api-access-zb7zg\") pod \"payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8\" (UID: \"9f850804-bb2e-4621-8b75-74cd586e59fa\") " pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:20:52.971118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.971095    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f850804-bb2e-4621-8b75-74cd586e59fa-plugins-config-volume\") pod \"payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8\" (UID: \"9f850804-bb2e-4621-8b75-74cd586e59fa\") " pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:20:52.971233 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.971207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4341db7b-e0b3-409f-9709-b30165676944-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n\" (UID: \"4341db7b-e0b3-409f-9709-b30165676944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:20:52.978969 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.978935    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8j4p\" (UniqueName: \"kubernetes.io/projected/4341db7b-e0b3-409f-9709-b30165676944-kube-api-access-n8j4p\") pod \"payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n\" (UID: \"4341db7b-e0b3-409f-9709-b30165676944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:20:52.979116 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:52.979103    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb7zg\" (UniqueName: \"kubernetes.io/projected/9f850804-bb2e-4621-8b75-74cd586e59fa-kube-api-access-zb7zg\") pod \"payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8\" (UID: \"9f850804-bb2e-4621-8b75-74cd586e59fa\") " pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:20:53.023617 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:53.023514    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:53.109735 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:53.109688    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:20:53.159326 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:53.159288    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:20:53.298404 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:53.298350    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"]
Sep 17 14:20:53.300438 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:20:53.300411    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04bde03_1a72_46d0_abd9_9ef395eda24d.slice/crio-c6de2c30f9ca2ba42ed7b67e79a0fe167d3b500135a7a9fdc9fee0357a6ea4f1 WatchSource:0}: Error finding container c6de2c30f9ca2ba42ed7b67e79a0fe167d3b500135a7a9fdc9fee0357a6ea4f1: Status 404 returned error can't find the container with id c6de2c30f9ca2ba42ed7b67e79a0fe167d3b500135a7a9fdc9fee0357a6ea4f1
Sep 17 14:20:53.391025 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:53.390994    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"]
Sep 17 14:20:53.392837 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:20:53.392809    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4341db7b_e0b3_409f_9709_b30165676944.slice/crio-0fd214d785d6e337ba6fc91e44a41564749926d51779b55bab32f8c02de87031 WatchSource:0}: Error finding container 0fd214d785d6e337ba6fc91e44a41564749926d51779b55bab32f8c02de87031: Status 404 returned error can't find the container with id 0fd214d785d6e337ba6fc91e44a41564749926d51779b55bab32f8c02de87031
Sep 17 14:20:53.444669 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:53.444523    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"]
Sep 17 14:20:53.447724 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:20:53.447695    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f850804_bb2e_4621_8b75_74cd586e59fa.slice/crio-257573d24f34ed5f5d7342fbb8d85a61c59531ccc26185b1babb8ae5e05970c7 WatchSource:0}: Error finding container 257573d24f34ed5f5d7342fbb8d85a61c59531ccc26185b1babb8ae5e05970c7: Status 404 returned error can't find the container with id 257573d24f34ed5f5d7342fbb8d85a61c59531ccc26185b1babb8ae5e05970c7
Sep 17 14:20:54.234191 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.234159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m" event={"ID":"b04bde03-1a72-46d0-abd9-9ef395eda24d","Type":"ContainerStarted","Data":"4fdb3b1c7e7cd606b60cffdc5b36ad745ecbd6f3b2fbbebf2fe8e7d650db996e"}
Sep 17 14:20:54.234666 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.234217    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:20:54.234666 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.234238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m" event={"ID":"b04bde03-1a72-46d0-abd9-9ef395eda24d","Type":"ContainerStarted","Data":"c6de2c30f9ca2ba42ed7b67e79a0fe167d3b500135a7a9fdc9fee0357a6ea4f1"}
Sep 17 14:20:54.235896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.235864    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n" event={"ID":"4341db7b-e0b3-409f-9709-b30165676944","Type":"ContainerStarted","Data":"78898f087a0386a98477a5c83e2177c0ae715338b4f2af63b0c7e95729b84b51"}
Sep 17 14:20:54.236021 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.235941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:20:54.236021 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.235970    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n" event={"ID":"4341db7b-e0b3-409f-9709-b30165676944","Type":"ContainerStarted","Data":"0fd214d785d6e337ba6fc91e44a41564749926d51779b55bab32f8c02de87031"}
Sep 17 14:20:54.237465 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.237444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8" event={"ID":"9f850804-bb2e-4621-8b75-74cd586e59fa","Type":"ContainerStarted","Data":"bd75c4d040ab121194120575649450b03ccbf6f8205878a9119f0cd846bcedfd"}
Sep 17 14:20:54.237572 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.237486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8" event={"ID":"9f850804-bb2e-4621-8b75-74cd586e59fa","Type":"ContainerStarted","Data":"257573d24f34ed5f5d7342fbb8d85a61c59531ccc26185b1babb8ae5e05970c7"}
Sep 17 14:20:54.237572 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.237516    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:20:54.252033 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.251972    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m" podStartSLOduration=1.7364807039999999 podStartE2EDuration="2.25194813s" totalImagesPullingTime="515.467426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:20:54.251497122 +0000 UTC m=+951.842672886" watchObservedRunningTime="2026-09-17 14:20:54.25194813 +0000 UTC m=+951.843123882"
Sep 17 14:20:54.272280 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.272206    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n" podStartSLOduration=2.272184315 podStartE2EDuration="2.272184315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:20:54.268490335 +0000 UTC m=+951.859666097" watchObservedRunningTime="2026-09-17 14:20:54.272184315 +0000 UTC m=+951.863360074"
Sep 17 14:20:54.294941 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:20:54.294887    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8" podStartSLOduration=2.294868206 podStartE2EDuration="2.294868206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:20:54.293880613 +0000 UTC m=+951.885056377" watchObservedRunningTime="2026-09-17 14:20:54.294868206 +0000 UTC m=+951.886043968"
Sep 17 14:21:00.249390 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:00.249342    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:21:04.303015 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:04.302970    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"]
Sep 17 14:21:04.303547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:04.303226    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225" podUID="f227571c-15b2-445b-80ff-9aa07b9b7a13" containerName="payload-processing" containerID="cri-o://ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11" gracePeriod=30
Sep 17 14:21:04.576410 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:04.576378    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:21:04.689042 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:04.689000    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f227571c-15b2-445b-80ff-9aa07b9b7a13-plugins-config-volume\") pod \"f227571c-15b2-445b-80ff-9aa07b9b7a13\" (UID: \"f227571c-15b2-445b-80ff-9aa07b9b7a13\") "
Sep 17 14:21:04.689242 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:04.689054    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm7tz\" (UniqueName: \"kubernetes.io/projected/f227571c-15b2-445b-80ff-9aa07b9b7a13-kube-api-access-zm7tz\") pod \"f227571c-15b2-445b-80ff-9aa07b9b7a13\" (UID: \"f227571c-15b2-445b-80ff-9aa07b9b7a13\") "
Sep 17 14:21:04.689573 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:04.689513    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f227571c-15b2-445b-80ff-9aa07b9b7a13-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f227571c-15b2-445b-80ff-9aa07b9b7a13" (UID: "f227571c-15b2-445b-80ff-9aa07b9b7a13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:21:04.691584 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:04.691551    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f227571c-15b2-445b-80ff-9aa07b9b7a13-kube-api-access-zm7tz" (OuterVolumeSpecName: "kube-api-access-zm7tz") pod "f227571c-15b2-445b-80ff-9aa07b9b7a13" (UID: "f227571c-15b2-445b-80ff-9aa07b9b7a13"). InnerVolumeSpecName "kube-api-access-zm7tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:21:04.790800 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:04.790762    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f227571c-15b2-445b-80ff-9aa07b9b7a13-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:21:04.790800 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:04.790793    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7tz\" (UniqueName: \"kubernetes.io/projected/f227571c-15b2-445b-80ff-9aa07b9b7a13-kube-api-access-zm7tz\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:21:05.246289 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.246258    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:21:05.246488 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.246341    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:21:05.311613 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.311562    2560 generic.go:356] "Generic (PLEG): container finished" podID="f227571c-15b2-445b-80ff-9aa07b9b7a13" containerID="ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11" exitCode=0
Sep 17 14:21:05.312075 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.311621    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"
Sep 17 14:21:05.312075 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.311638    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225" event={"ID":"f227571c-15b2-445b-80ff-9aa07b9b7a13","Type":"ContainerDied","Data":"ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11"}
Sep 17 14:21:05.312075 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.311725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225" event={"ID":"f227571c-15b2-445b-80ff-9aa07b9b7a13","Type":"ContainerDied","Data":"16430a149a119d2d0329bdb80e5d032550295ed09452f350c096d1055762dccf"}
Sep 17 14:21:05.312075 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.311749    2560 scope.go:117] "RemoveContainer" containerID="ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11"
Sep 17 14:21:05.322271 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.322239    2560 scope.go:117] "RemoveContainer" containerID="ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11"
Sep 17 14:21:05.322764 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:21:05.322742    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11\": container with ID starting with ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11 not found: ID does not exist" containerID="ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11"
Sep 17 14:21:05.322819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.322777    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11"} err="failed to get container status \"ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11\": rpc error: code = NotFound desc = could not find container \"ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11\": container with ID starting with ab2c1eb885c9b3de06622db564e294849f1066d5ad66eff2f38cb0f4cf322d11 not found: ID does not exist"
Sep 17 14:21:05.328507 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.328474    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"]
Sep 17 14:21:05.333378 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:05.333327    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3f6f32-76c5c496b9-cl225"]
Sep 17 14:21:06.966448 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:06.966417    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f227571c-15b2-445b-80ff-9aa07b9b7a13" path="/var/lib/kubelet/pods/f227571c-15b2-445b-80ff-9aa07b9b7a13/volumes"
Sep 17 14:21:18.178331 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.178294    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"]
Sep 17 14:21:18.178866 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.178568    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m" podUID="b04bde03-1a72-46d0-abd9-9ef395eda24d" containerName="maas-api" containerID="cri-o://4fdb3b1c7e7cd606b60cffdc5b36ad745ecbd6f3b2fbbebf2fe8e7d650db996e" gracePeriod=30
Sep 17 14:21:18.222147 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.222109    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"]
Sep 17 14:21:18.222408 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.222340    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8" podUID="9f850804-bb2e-4621-8b75-74cd586e59fa" containerName="payload-processing" containerID="cri-o://bd75c4d040ab121194120575649450b03ccbf6f8205878a9119f0cd846bcedfd" gracePeriod=30
Sep 17 14:21:18.228575 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.228508    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"]
Sep 17 14:21:18.229216 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.229153    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n" podUID="4341db7b-e0b3-409f-9709-b30165676944" containerName="payload-pre-processing" containerID="cri-o://78898f087a0386a98477a5c83e2177c0ae715338b4f2af63b0c7e95729b84b51" gracePeriod=30
Sep 17 14:21:18.384677 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.384625    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f850804-bb2e-4621-8b75-74cd586e59fa" containerID="bd75c4d040ab121194120575649450b03ccbf6f8205878a9119f0cd846bcedfd" exitCode=0
Sep 17 14:21:18.384872 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.384756    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8" event={"ID":"9f850804-bb2e-4621-8b75-74cd586e59fa","Type":"ContainerDied","Data":"bd75c4d040ab121194120575649450b03ccbf6f8205878a9119f0cd846bcedfd"}
Sep 17 14:21:18.387553 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.387510    2560 generic.go:356] "Generic (PLEG): container finished" podID="b04bde03-1a72-46d0-abd9-9ef395eda24d" containerID="4fdb3b1c7e7cd606b60cffdc5b36ad745ecbd6f3b2fbbebf2fe8e7d650db996e" exitCode=0
Sep 17 14:21:18.387738 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.387699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m" event={"ID":"b04bde03-1a72-46d0-abd9-9ef395eda24d","Type":"ContainerDied","Data":"4fdb3b1c7e7cd606b60cffdc5b36ad745ecbd6f3b2fbbebf2fe8e7d650db996e"}
Sep 17 14:21:18.390214 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.390175    2560 generic.go:356] "Generic (PLEG): container finished" podID="4341db7b-e0b3-409f-9709-b30165676944" containerID="78898f087a0386a98477a5c83e2177c0ae715338b4f2af63b0c7e95729b84b51" exitCode=0
Sep 17 14:21:18.390417 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.390273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n" event={"ID":"4341db7b-e0b3-409f-9709-b30165676944","Type":"ContainerDied","Data":"78898f087a0386a98477a5c83e2177c0ae715338b4f2af63b0c7e95729b84b51"}
Sep 17 14:21:18.499632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.499599    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:21:18.546903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.545495    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qpbs\" (UniqueName: \"kubernetes.io/projected/b04bde03-1a72-46d0-abd9-9ef395eda24d-kube-api-access-9qpbs\") pod \"b04bde03-1a72-46d0-abd9-9ef395eda24d\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") "
Sep 17 14:21:18.546903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.546004    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-maas-api-tls\") pod \"b04bde03-1a72-46d0-abd9-9ef395eda24d\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") "
Sep 17 14:21:18.546903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.546556    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-metrics-certs\") pod \"b04bde03-1a72-46d0-abd9-9ef395eda24d\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") "
Sep 17 14:21:18.546903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.546658    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04bde03-1a72-46d0-abd9-9ef395eda24d-openshift-service-ca\") pod \"b04bde03-1a72-46d0-abd9-9ef395eda24d\" (UID: \"b04bde03-1a72-46d0-abd9-9ef395eda24d\") "
Sep 17 14:21:18.547276 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.547034    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04bde03-1a72-46d0-abd9-9ef395eda24d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b04bde03-1a72-46d0-abd9-9ef395eda24d" (UID: "b04bde03-1a72-46d0-abd9-9ef395eda24d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:21:18.551916 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.547499    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04bde03-1a72-46d0-abd9-9ef395eda24d-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.551916 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.551079    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04bde03-1a72-46d0-abd9-9ef395eda24d-kube-api-access-9qpbs" (OuterVolumeSpecName: "kube-api-access-9qpbs") pod "b04bde03-1a72-46d0-abd9-9ef395eda24d" (UID: "b04bde03-1a72-46d0-abd9-9ef395eda24d"). InnerVolumeSpecName "kube-api-access-9qpbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:21:18.552868 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.552812    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b04bde03-1a72-46d0-abd9-9ef395eda24d" (UID: "b04bde03-1a72-46d0-abd9-9ef395eda24d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:21:18.554922 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.554869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b04bde03-1a72-46d0-abd9-9ef395eda24d" (UID: "b04bde03-1a72-46d0-abd9-9ef395eda24d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:21:18.575248 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.575220    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:21:18.579170 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.579142    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:21:18.648090 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.648042    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f850804-bb2e-4621-8b75-74cd586e59fa-plugins-config-volume\") pod \"9f850804-bb2e-4621-8b75-74cd586e59fa\" (UID: \"9f850804-bb2e-4621-8b75-74cd586e59fa\") "
Sep 17 14:21:18.648301 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.648122    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8j4p\" (UniqueName: \"kubernetes.io/projected/4341db7b-e0b3-409f-9709-b30165676944-kube-api-access-n8j4p\") pod \"4341db7b-e0b3-409f-9709-b30165676944\" (UID: \"4341db7b-e0b3-409f-9709-b30165676944\") "
Sep 17 14:21:18.648301 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.648187    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb7zg\" (UniqueName: \"kubernetes.io/projected/9f850804-bb2e-4621-8b75-74cd586e59fa-kube-api-access-zb7zg\") pod \"9f850804-bb2e-4621-8b75-74cd586e59fa\" (UID: \"9f850804-bb2e-4621-8b75-74cd586e59fa\") "
Sep 17 14:21:18.648458 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.648312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4341db7b-e0b3-409f-9709-b30165676944-plugins-config-volume\") pod \"4341db7b-e0b3-409f-9709-b30165676944\" (UID: \"4341db7b-e0b3-409f-9709-b30165676944\") "
Sep 17 14:21:18.648629 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.648553    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f850804-bb2e-4621-8b75-74cd586e59fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f850804-bb2e-4621-8b75-74cd586e59fa" (UID: "9f850804-bb2e-4621-8b75-74cd586e59fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:21:18.648722 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.648701    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.648807 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.648719    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qpbs\" (UniqueName: \"kubernetes.io/projected/b04bde03-1a72-46d0-abd9-9ef395eda24d-kube-api-access-9qpbs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.648807 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.648728    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4341db7b-e0b3-409f-9709-b30165676944-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4341db7b-e0b3-409f-9709-b30165676944" (UID: "4341db7b-e0b3-409f-9709-b30165676944"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:21:18.648807 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.648736    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f850804-bb2e-4621-8b75-74cd586e59fa-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.648807 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.648781    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04bde03-1a72-46d0-abd9-9ef395eda24d-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.650861 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.650830    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4341db7b-e0b3-409f-9709-b30165676944-kube-api-access-n8j4p" (OuterVolumeSpecName: "kube-api-access-n8j4p") pod "4341db7b-e0b3-409f-9709-b30165676944" (UID: "4341db7b-e0b3-409f-9709-b30165676944"). InnerVolumeSpecName "kube-api-access-n8j4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:21:18.650959 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.650856    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f850804-bb2e-4621-8b75-74cd586e59fa-kube-api-access-zb7zg" (OuterVolumeSpecName: "kube-api-access-zb7zg") pod "9f850804-bb2e-4621-8b75-74cd586e59fa" (UID: "9f850804-bb2e-4621-8b75-74cd586e59fa"). InnerVolumeSpecName "kube-api-access-zb7zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:21:18.750374 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.750327    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8j4p\" (UniqueName: \"kubernetes.io/projected/4341db7b-e0b3-409f-9709-b30165676944-kube-api-access-n8j4p\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.750559 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.750390    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb7zg\" (UniqueName: \"kubernetes.io/projected/9f850804-bb2e-4621-8b75-74cd586e59fa-kube-api-access-zb7zg\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.750559 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:18.750405    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4341db7b-e0b3-409f-9709-b30165676944-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:21:19.398147 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.398101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m" event={"ID":"b04bde03-1a72-46d0-abd9-9ef395eda24d","Type":"ContainerDied","Data":"c6de2c30f9ca2ba42ed7b67e79a0fe167d3b500135a7a9fdc9fee0357a6ea4f1"}
Sep 17 14:21:19.398663 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.398181    2560 scope.go:117] "RemoveContainer" containerID="4fdb3b1c7e7cd606b60cffdc5b36ad745ecbd6f3b2fbbebf2fe8e7d650db996e"
Sep 17 14:21:19.398663 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.398111    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"
Sep 17 14:21:19.400204 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.400182    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"
Sep 17 14:21:19.400316 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.400243    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n" event={"ID":"4341db7b-e0b3-409f-9709-b30165676944","Type":"ContainerDied","Data":"0fd214d785d6e337ba6fc91e44a41564749926d51779b55bab32f8c02de87031"}
Sep 17 14:21:19.403169 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.402525    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8" event={"ID":"9f850804-bb2e-4621-8b75-74cd586e59fa","Type":"ContainerDied","Data":"257573d24f34ed5f5d7342fbb8d85a61c59531ccc26185b1babb8ae5e05970c7"}
Sep 17 14:21:19.403169 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.402569    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"
Sep 17 14:21:19.410862 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.410843    2560 scope.go:117] "RemoveContainer" containerID="78898f087a0386a98477a5c83e2177c0ae715338b4f2af63b0c7e95729b84b51"
Sep 17 14:21:19.415864 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.415827    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"]
Sep 17 14:21:19.420127 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.420093    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99500ae5-6678978f75-plr8m"]
Sep 17 14:21:19.421512 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.421491    2560 scope.go:117] "RemoveContainer" containerID="bd75c4d040ab121194120575649450b03ccbf6f8205878a9119f0cd846bcedfd"
Sep 17 14:21:19.437937 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.437902    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"]
Sep 17 14:21:19.464490 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.464405    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99500ae5-7c44c7d77d-b7f6n"]
Sep 17 14:21:19.494086 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.494048    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"]
Sep 17 14:21:19.505173 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:19.505143    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99500ae5-9f5c78649-dgdp8"]
Sep 17 14:21:20.965575 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:20.965539    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4341db7b-e0b3-409f-9709-b30165676944" path="/var/lib/kubelet/pods/4341db7b-e0b3-409f-9709-b30165676944/volumes"
Sep 17 14:21:20.965960 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:20.965936    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f850804-bb2e-4621-8b75-74cd586e59fa" path="/var/lib/kubelet/pods/9f850804-bb2e-4621-8b75-74cd586e59fa/volumes"
Sep 17 14:21:20.966285 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:21:20.966272    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b04bde03-1a72-46d0-abd9-9ef395eda24d" path="/var/lib/kubelet/pods/b04bde03-1a72-46d0-abd9-9ef395eda24d/volumes"
Sep 17 14:22:09.119158 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.119115    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"]
Sep 17 14:22:09.119797 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.119771    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4341db7b-e0b3-409f-9709-b30165676944" containerName="payload-pre-processing"
Sep 17 14:22:09.119873 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.119802    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4341db7b-e0b3-409f-9709-b30165676944" containerName="payload-pre-processing"
Sep 17 14:22:09.119873 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.119825    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f227571c-15b2-445b-80ff-9aa07b9b7a13" containerName="payload-processing"
Sep 17 14:22:09.119873 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.119835    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f227571c-15b2-445b-80ff-9aa07b9b7a13" containerName="payload-processing"
Sep 17 14:22:09.119873 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.119855    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f850804-bb2e-4621-8b75-74cd586e59fa" containerName="payload-processing"
Sep 17 14:22:09.119873 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.119865    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f850804-bb2e-4621-8b75-74cd586e59fa" containerName="payload-processing"
Sep 17 14:22:09.120131 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.119876    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b04bde03-1a72-46d0-abd9-9ef395eda24d" containerName="maas-api"
Sep 17 14:22:09.120131 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.119886    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b04bde03-1a72-46d0-abd9-9ef395eda24d" containerName="maas-api"
Sep 17 14:22:09.120131 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.120007    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f227571c-15b2-445b-80ff-9aa07b9b7a13" containerName="payload-processing"
Sep 17 14:22:09.120131 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.120024    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4341db7b-e0b3-409f-9709-b30165676944" containerName="payload-pre-processing"
Sep 17 14:22:09.120131 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.120033    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b04bde03-1a72-46d0-abd9-9ef395eda24d" containerName="maas-api"
Sep 17 14:22:09.120131 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.120045    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f850804-bb2e-4621-8b75-74cd586e59fa" containerName="payload-processing"
Sep 17 14:22:09.125072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.125044    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.128104 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.128077    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-68183c-serving-cert\""
Sep 17 14:22:09.134886 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.134853    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"]
Sep 17 14:22:09.170048 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.170014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-maas-api-tls\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.170268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.170078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-metrics-certs\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.170268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.170197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtsw\" (UniqueName: \"kubernetes.io/projected/1e7b9a88-abe7-4f12-b2de-275d504fc326-kube-api-access-gxtsw\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.170395 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.170295    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e7b9a88-abe7-4f12-b2de-275d504fc326-openshift-service-ca\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.271573 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.271527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e7b9a88-abe7-4f12-b2de-275d504fc326-openshift-service-ca\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.271573 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.271581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-maas-api-tls\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.271850 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.271645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-metrics-certs\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.271850 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.271712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtsw\" (UniqueName: \"kubernetes.io/projected/1e7b9a88-abe7-4f12-b2de-275d504fc326-kube-api-access-gxtsw\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.271850 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:22:09.271735    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-serving-cert: secret "maas-api-e2e-explicit-68183c-serving-cert" not found
Sep 17 14:22:09.271850 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:22:09.271809    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-maas-api-tls podName:1e7b9a88-abe7-4f12-b2de-275d504fc326 nodeName:}" failed. No retries permitted until 2026-09-17 14:22:09.771792049 +0000 UTC m=+1027.362967789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-maas-api-tls") pod "maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l" (UID: "1e7b9a88-abe7-4f12-b2de-275d504fc326") : secret "maas-api-e2e-explicit-68183c-serving-cert" not found
Sep 17 14:22:09.272158 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.272138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e7b9a88-abe7-4f12-b2de-275d504fc326-openshift-service-ca\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.274177 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.274153    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-metrics-certs\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.283194 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.283163    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtsw\" (UniqueName: \"kubernetes.io/projected/1e7b9a88-abe7-4f12-b2de-275d504fc326-kube-api-access-gxtsw\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.764889 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.764850    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"]
Sep 17 14:22:09.768507 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.768476    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:22:09.771204 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.771168    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-68183c\""
Sep 17 14:22:09.771204 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.771192    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-68183c-dockercfg-52dds\""
Sep 17 14:22:09.775631 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.775600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-maas-api-tls\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.778719 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.778691    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-maas-api-tls\") pod \"maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:09.779794 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.779769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"]
Sep 17 14:22:09.876613 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.876577    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af4018b-d13e-41ac-9f84-762464587829-plugins-config-volume\") pod \"payload-processing-e2e-explicit-68183c-8c86449-cd6kd\" (UID: \"2af4018b-d13e-41ac-9f84-762464587829\") " pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:22:09.876807 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.876749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwcv2\" (UniqueName: \"kubernetes.io/projected/2af4018b-d13e-41ac-9f84-762464587829-kube-api-access-gwcv2\") pod \"payload-processing-e2e-explicit-68183c-8c86449-cd6kd\" (UID: \"2af4018b-d13e-41ac-9f84-762464587829\") " pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:22:09.977754 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.977708    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwcv2\" (UniqueName: \"kubernetes.io/projected/2af4018b-d13e-41ac-9f84-762464587829-kube-api-access-gwcv2\") pod \"payload-processing-e2e-explicit-68183c-8c86449-cd6kd\" (UID: \"2af4018b-d13e-41ac-9f84-762464587829\") " pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:22:09.977954 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.977783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af4018b-d13e-41ac-9f84-762464587829-plugins-config-volume\") pod \"payload-processing-e2e-explicit-68183c-8c86449-cd6kd\" (UID: \"2af4018b-d13e-41ac-9f84-762464587829\") " pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:22:09.978577 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.978557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af4018b-d13e-41ac-9f84-762464587829-plugins-config-volume\") pod \"payload-processing-e2e-explicit-68183c-8c86449-cd6kd\" (UID: \"2af4018b-d13e-41ac-9f84-762464587829\") " pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:22:09.987119 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:09.987086    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwcv2\" (UniqueName: \"kubernetes.io/projected/2af4018b-d13e-41ac-9f84-762464587829-kube-api-access-gwcv2\") pod \"payload-processing-e2e-explicit-68183c-8c86449-cd6kd\" (UID: \"2af4018b-d13e-41ac-9f84-762464587829\") " pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:22:10.038619 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:10.038526    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:10.080751 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:10.080712    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:22:10.319631 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:10.319599    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"]
Sep 17 14:22:10.322135 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:22:10.322102    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7b9a88_abe7_4f12_b2de_275d504fc326.slice/crio-a80a18a759be19bce7d1799b06eb84012e447c05f7ef6fd12aa9209ec56f8efe WatchSource:0}: Error finding container a80a18a759be19bce7d1799b06eb84012e447c05f7ef6fd12aa9209ec56f8efe: Status 404 returned error can't find the container with id a80a18a759be19bce7d1799b06eb84012e447c05f7ef6fd12aa9209ec56f8efe
Sep 17 14:22:10.323998 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:10.323975    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:22:10.377985 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:10.377954    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"]
Sep 17 14:22:10.380762 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:22:10.380731    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af4018b_d13e_41ac_9f84_762464587829.slice/crio-3550e740cdecc1e063f4ec606acd87ad14d6894ddcbef2e72dee05cbf3bcc47e WatchSource:0}: Error finding container 3550e740cdecc1e063f4ec606acd87ad14d6894ddcbef2e72dee05cbf3bcc47e: Status 404 returned error can't find the container with id 3550e740cdecc1e063f4ec606acd87ad14d6894ddcbef2e72dee05cbf3bcc47e
Sep 17 14:22:10.661803 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:10.661707    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l" event={"ID":"1e7b9a88-abe7-4f12-b2de-275d504fc326","Type":"ContainerStarted","Data":"a80a18a759be19bce7d1799b06eb84012e447c05f7ef6fd12aa9209ec56f8efe"}
Sep 17 14:22:10.663378 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:10.663336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd" event={"ID":"2af4018b-d13e-41ac-9f84-762464587829","Type":"ContainerStarted","Data":"bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a"}
Sep 17 14:22:10.663528 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:10.663406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd" event={"ID":"2af4018b-d13e-41ac-9f84-762464587829","Type":"ContainerStarted","Data":"3550e740cdecc1e063f4ec606acd87ad14d6894ddcbef2e72dee05cbf3bcc47e"}
Sep 17 14:22:10.663528 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:10.663431    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:22:10.683867 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:10.683803    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd" podStartSLOduration=1.683781266 podStartE2EDuration="1.683781266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:10.68183049 +0000 UTC m=+1028.273006253" watchObservedRunningTime="2026-09-17 14:22:10.683781266 +0000 UTC m=+1028.274957036"
Sep 17 14:22:11.670153 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:11.670117    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l" event={"ID":"1e7b9a88-abe7-4f12-b2de-275d504fc326","Type":"ContainerStarted","Data":"221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6"}
Sep 17 14:22:11.670625 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:11.670245    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:11.689133 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:11.689074    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l" podStartSLOduration=2.272691639 podStartE2EDuration="2.689056911s" totalImagesPullingTime="416.365272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:11.687857622 +0000 UTC m=+1029.279033394" watchObservedRunningTime="2026-09-17 14:22:11.689056911 +0000 UTC m=+1029.280232672"
Sep 17 14:22:17.681215 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:17.681183    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:22:18.090131 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.090093    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"]
Sep 17 14:22:18.093838 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.093809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.096201 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.096178    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1269b2\"/\"premium-w0-1269b2-kserve-self-signed-certs\""
Sep 17 14:22:18.102317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.102287    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"]
Sep 17 14:22:18.165225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.165190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tls-certs\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.165439 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.165238    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-model-cache\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.165439 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.165266    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tmp-dir\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.165439 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.165284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdbb4\" (UniqueName: \"kubernetes.io/projected/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-kube-api-access-xdbb4\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.165439 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.165371    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-dshm\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.165674 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.165475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-home\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.266170 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.266125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-home\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.266170 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.266171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tls-certs\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.266453 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.266423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-model-cache\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.266513 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.266497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tmp-dir\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.266553 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.266530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdbb4\" (UniqueName: \"kubernetes.io/projected/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-kube-api-access-xdbb4\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.266607 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.266592    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-dshm\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.266666 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.266618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-home\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.266858 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.266837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tmp-dir\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.266930 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.266833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-model-cache\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.268643 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.268622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-dshm\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.268820 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.268802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tls-certs\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.276239 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.276211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdbb4\" (UniqueName: \"kubernetes.io/projected/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-kube-api-access-xdbb4\") pod \"premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") " pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.403274 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.403176    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:18.676426 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.676396    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"]
Sep 17 14:22:18.679141 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:22:18.679112    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6d5aab3_e6bc_4c69_9a1d_25c6bf40a59c.slice/crio-495ebee358028ef1b0a393590434a55f39d1eb56b0f71715ddf3cf57fc0b21f8 WatchSource:0}: Error finding container 495ebee358028ef1b0a393590434a55f39d1eb56b0f71715ddf3cf57fc0b21f8: Status 404 returned error can't find the container with id 495ebee358028ef1b0a393590434a55f39d1eb56b0f71715ddf3cf57fc0b21f8
Sep 17 14:22:18.704687 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:18.704653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f" event={"ID":"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c","Type":"ContainerStarted","Data":"495ebee358028ef1b0a393590434a55f39d1eb56b0f71715ddf3cf57fc0b21f8"}
Sep 17 14:22:19.711191 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:19.711151    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f" event={"ID":"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c","Type":"ContainerStarted","Data":"8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a"}
Sep 17 14:22:19.731379 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:19.731315    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f" podStartSLOduration=1.731296717 podStartE2EDuration="1.731296717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:19.729537551 +0000 UTC m=+1037.320713318" watchObservedRunningTime="2026-09-17 14:22:19.731296717 +0000 UTC m=+1037.322472513"
Sep 17 14:22:21.673093 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:21.673062    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:22:28.403619 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:28.403566    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:28.404219 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:28.403643    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:28.416811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:28.416770    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:22:28.770291 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:22:28.770261    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:23:05.623233 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.623138    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"]
Sep 17 14:23:05.626707 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.626538    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.629368 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.629332    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2137c903-serving-cert\""
Sep 17 14:23:05.637463 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.637435    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"]
Sep 17 14:23:05.658932 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.658898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-openshift-service-ca\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.658932 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.658935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-maas-api-tls\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.659201 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.658993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-metrics-certs\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.659201 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.659138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ct5\" (UniqueName: \"kubernetes.io/projected/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-kube-api-access-87ct5\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.760311 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.760271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-openshift-service-ca\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.760311 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.760313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-maas-api-tls\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.760551 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.760374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-metrics-certs\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.760551 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.760434    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ct5\" (UniqueName: \"kubernetes.io/projected/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-kube-api-access-87ct5\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.761033 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.761010    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-openshift-service-ca\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.762874 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.762850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-maas-api-tls\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.762973 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.762905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-metrics-certs\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.769811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.769778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ct5\" (UniqueName: \"kubernetes.io/projected/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-kube-api-access-87ct5\") pod \"maas-api-e2e-mt-2137c903-875854589-vnsjp\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:05.939753 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:05.939663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:06.219511 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:06.219481    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"]
Sep 17 14:23:06.221662 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:23:06.221626    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844bf3cb_8aaf_41d8_9928_49709d8f0ad0.slice/crio-d04f4577bd4769ae9148ac91ab1fd4697ce00e46012dd105f839944047e20fbf WatchSource:0}: Error finding container d04f4577bd4769ae9148ac91ab1fd4697ce00e46012dd105f839944047e20fbf: Status 404 returned error can't find the container with id d04f4577bd4769ae9148ac91ab1fd4697ce00e46012dd105f839944047e20fbf
Sep 17 14:23:06.968276 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:06.968184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp" event={"ID":"844bf3cb-8aaf-41d8-9928-49709d8f0ad0","Type":"ContainerStarted","Data":"c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8"}
Sep 17 14:23:06.968276 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:06.968256    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:06.968681 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:06.968282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp" event={"ID":"844bf3cb-8aaf-41d8-9928-49709d8f0ad0","Type":"ContainerStarted","Data":"d04f4577bd4769ae9148ac91ab1fd4697ce00e46012dd105f839944047e20fbf"}
Sep 17 14:23:06.999861 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:06.999785    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp" podStartSLOduration=1.5645515890000001 podStartE2EDuration="1.999761065s" totalImagesPullingTime="435.209476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:23:06.997182104 +0000 UTC m=+1084.588357867" watchObservedRunningTime="2026-09-17 14:23:06.999761065 +0000 UTC m=+1084.590936830"
Sep 17 14:23:12.978864 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:12.978834    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:19.171883 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.171839    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"]
Sep 17 14:23:19.177107 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.177080    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.179769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.179737    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-68183c\"/\"kube-root-ca.crt\""
Sep 17 14:23:19.181174 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.181150    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-68183c\"/\"default-dockercfg-tjwwr\""
Sep 17 14:23:19.181174 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.181164    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-68183c\"/\"openshift-service-ca.crt\""
Sep 17 14:23:19.181369 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.181151    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-68183c\"/\"explicit-model-68183c-kserve-self-signed-certs\""
Sep 17 14:23:19.188804 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.188766    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"]
Sep 17 14:23:19.289666 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.289622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-tmp-dir\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.289666 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.289668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c876472f-197b-4c1b-8ed8-e03047bf64ec-tls-certs\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.289903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.289704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-dshm\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.289903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.289812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-home\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.289903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.289877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pz5g\" (UniqueName: \"kubernetes.io/projected/c876472f-197b-4c1b-8ed8-e03047bf64ec-kube-api-access-8pz5g\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.290050 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.289906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-model-cache\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.390859 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.390816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-dshm\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.391072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.390887    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-home\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.391072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.390927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pz5g\" (UniqueName: \"kubernetes.io/projected/c876472f-197b-4c1b-8ed8-e03047bf64ec-kube-api-access-8pz5g\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.391072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.390949    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-model-cache\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.391072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.390996    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-tmp-dir\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.391072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.391020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c876472f-197b-4c1b-8ed8-e03047bf64ec-tls-certs\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.391385 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.391339    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-model-cache\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.391533 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.391514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-home\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.391602 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.391540    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-tmp-dir\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.393225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.393196    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-dshm\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.393420 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.393403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c876472f-197b-4c1b-8ed8-e03047bf64ec-tls-certs\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.402146 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.402116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pz5g\" (UniqueName: \"kubernetes.io/projected/c876472f-197b-4c1b-8ed8-e03047bf64ec-kube-api-access-8pz5g\") pod \"explicit-model-68183c-kserve-8646c6c9ff-x2d5l\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") " pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.490585 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.490542    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:19.765777 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:19.765749    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"]
Sep 17 14:23:19.768011 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:23:19.767986    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc876472f_197b_4c1b_8ed8_e03047bf64ec.slice/crio-5c4f1d5ee841471726fc4331f3658e4e0f9039255799096e27fd161bef8bd729 WatchSource:0}: Error finding container 5c4f1d5ee841471726fc4331f3658e4e0f9039255799096e27fd161bef8bd729: Status 404 returned error can't find the container with id 5c4f1d5ee841471726fc4331f3658e4e0f9039255799096e27fd161bef8bd729
Sep 17 14:23:20.037574 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:20.037479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l" event={"ID":"c876472f-197b-4c1b-8ed8-e03047bf64ec","Type":"ContainerStarted","Data":"e33960d86461b3ca7eaf8f5856c19cb4ed6ccee5f0af992694c1357f0052d32a"}
Sep 17 14:23:20.037574 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:20.037527    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l" event={"ID":"c876472f-197b-4c1b-8ed8-e03047bf64ec","Type":"ContainerStarted","Data":"5c4f1d5ee841471726fc4331f3658e4e0f9039255799096e27fd161bef8bd729"}
Sep 17 14:23:20.059638 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:20.059571    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l" podStartSLOduration=1.059552082 podStartE2EDuration="1.059552082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:23:20.059425226 +0000 UTC m=+1097.650600987" watchObservedRunningTime="2026-09-17 14:23:20.059552082 +0000 UTC m=+1097.650727847"
Sep 17 14:23:29.491403 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:29.491291    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:29.491403 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:29.491339    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:29.504787 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:29.504758    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:30.105056 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:30.105025    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:23:50.179215 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.179176    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"]
Sep 17 14:23:50.179719 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.179467    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp" podUID="844bf3cb-8aaf-41d8-9928-49709d8f0ad0" containerName="maas-api" containerID="cri-o://c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8" gracePeriod=30
Sep 17 14:23:50.436391 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.436309    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:50.522206 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.522171    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87ct5\" (UniqueName: \"kubernetes.io/projected/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-kube-api-access-87ct5\") pod \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") "
Sep 17 14:23:50.522393 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.522260    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-openshift-service-ca\") pod \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") "
Sep 17 14:23:50.522441 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.522394    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-metrics-certs\") pod \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") "
Sep 17 14:23:50.522441 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.522421    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-maas-api-tls\") pod \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\" (UID: \"844bf3cb-8aaf-41d8-9928-49709d8f0ad0\") "
Sep 17 14:23:50.522658 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.522632    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "844bf3cb-8aaf-41d8-9928-49709d8f0ad0" (UID: "844bf3cb-8aaf-41d8-9928-49709d8f0ad0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:23:50.522769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.522723    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:23:50.524281 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.524258    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-kube-api-access-87ct5" (OuterVolumeSpecName: "kube-api-access-87ct5") pod "844bf3cb-8aaf-41d8-9928-49709d8f0ad0" (UID: "844bf3cb-8aaf-41d8-9928-49709d8f0ad0"). InnerVolumeSpecName "kube-api-access-87ct5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:23:50.524465 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.524447    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "844bf3cb-8aaf-41d8-9928-49709d8f0ad0" (UID: "844bf3cb-8aaf-41d8-9928-49709d8f0ad0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:23:50.524545 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.524460    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "844bf3cb-8aaf-41d8-9928-49709d8f0ad0" (UID: "844bf3cb-8aaf-41d8-9928-49709d8f0ad0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:23:50.623317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.623277    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:23:50.623317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.623311    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:23:50.623317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:50.623320    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ct5\" (UniqueName: \"kubernetes.io/projected/844bf3cb-8aaf-41d8-9928-49709d8f0ad0-kube-api-access-87ct5\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:23:51.203608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:51.203569    2560 generic.go:356] "Generic (PLEG): container finished" podID="844bf3cb-8aaf-41d8-9928-49709d8f0ad0" containerID="c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8" exitCode=0
Sep 17 14:23:51.204044 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:51.203622    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"
Sep 17 14:23:51.204044 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:51.203641    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp" event={"ID":"844bf3cb-8aaf-41d8-9928-49709d8f0ad0","Type":"ContainerDied","Data":"c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8"}
Sep 17 14:23:51.204044 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:51.203702    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp" event={"ID":"844bf3cb-8aaf-41d8-9928-49709d8f0ad0","Type":"ContainerDied","Data":"d04f4577bd4769ae9148ac91ab1fd4697ce00e46012dd105f839944047e20fbf"}
Sep 17 14:23:51.204044 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:51.203726    2560 scope.go:117] "RemoveContainer" containerID="c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8"
Sep 17 14:23:51.213223 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:51.213199    2560 scope.go:117] "RemoveContainer" containerID="c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8"
Sep 17 14:23:51.213563 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:23:51.213541    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8\": container with ID starting with c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8 not found: ID does not exist" containerID="c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8"
Sep 17 14:23:51.213633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:51.213572    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8"} err="failed to get container status \"c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8\": rpc error: code = NotFound desc = could not find container \"c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8\": container with ID starting with c8c0eb8a6bcceeb6dc139578604a47dbcaa34cb54ef1d5d86186b037234555e8 not found: ID does not exist"
Sep 17 14:23:51.224336 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:51.224238    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"]
Sep 17 14:23:51.229956 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:51.229926    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2137c903-875854589-vnsjp"]
Sep 17 14:23:52.966062 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:52.966022    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="844bf3cb-8aaf-41d8-9928-49709d8f0ad0" path="/var/lib/kubelet/pods/844bf3cb-8aaf-41d8-9928-49709d8f0ad0/volumes"
Sep 17 14:23:59.073353 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:23:59.073313    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"]
Sep 17 14:24:04.559516 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.559480    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"]
Sep 17 14:24:04.560016 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.559709    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l" podUID="1e7b9a88-abe7-4f12-b2de-275d504fc326" containerName="maas-api" containerID="cri-o://221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6" gracePeriod=30
Sep 17 14:24:04.614197 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.614159    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"]
Sep 17 14:24:04.614490 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.614461    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd" podUID="2af4018b-d13e-41ac-9f84-762464587829" containerName="payload-processing" containerID="cri-o://bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a" gracePeriod=30
Sep 17 14:24:04.898763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.898728    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:24:04.901575 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.901551    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:24:04.975170 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.975137    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-maas-api-tls\") pod \"1e7b9a88-abe7-4f12-b2de-275d504fc326\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") "
Sep 17 14:24:04.975390 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.975177    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxtsw\" (UniqueName: \"kubernetes.io/projected/1e7b9a88-abe7-4f12-b2de-275d504fc326-kube-api-access-gxtsw\") pod \"1e7b9a88-abe7-4f12-b2de-275d504fc326\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") "
Sep 17 14:24:04.975390 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.975198    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e7b9a88-abe7-4f12-b2de-275d504fc326-openshift-service-ca\") pod \"1e7b9a88-abe7-4f12-b2de-275d504fc326\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") "
Sep 17 14:24:04.975390 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.975222    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwcv2\" (UniqueName: \"kubernetes.io/projected/2af4018b-d13e-41ac-9f84-762464587829-kube-api-access-gwcv2\") pod \"2af4018b-d13e-41ac-9f84-762464587829\" (UID: \"2af4018b-d13e-41ac-9f84-762464587829\") "
Sep 17 14:24:04.975390 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.975300    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-metrics-certs\") pod \"1e7b9a88-abe7-4f12-b2de-275d504fc326\" (UID: \"1e7b9a88-abe7-4f12-b2de-275d504fc326\") "
Sep 17 14:24:04.975390 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.975330    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af4018b-d13e-41ac-9f84-762464587829-plugins-config-volume\") pod \"2af4018b-d13e-41ac-9f84-762464587829\" (UID: \"2af4018b-d13e-41ac-9f84-762464587829\") "
Sep 17 14:24:04.975937 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.975905    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7b9a88-abe7-4f12-b2de-275d504fc326-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e7b9a88-abe7-4f12-b2de-275d504fc326" (UID: "1e7b9a88-abe7-4f12-b2de-275d504fc326"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:04.976175 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.976148    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af4018b-d13e-41ac-9f84-762464587829-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2af4018b-d13e-41ac-9f84-762464587829" (UID: "2af4018b-d13e-41ac-9f84-762464587829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:04.977707 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.977682    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af4018b-d13e-41ac-9f84-762464587829-kube-api-access-gwcv2" (OuterVolumeSpecName: "kube-api-access-gwcv2") pod "2af4018b-d13e-41ac-9f84-762464587829" (UID: "2af4018b-d13e-41ac-9f84-762464587829"). InnerVolumeSpecName "kube-api-access-gwcv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:24:04.977817 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.977718    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7b9a88-abe7-4f12-b2de-275d504fc326-kube-api-access-gxtsw" (OuterVolumeSpecName: "kube-api-access-gxtsw") pod "1e7b9a88-abe7-4f12-b2de-275d504fc326" (UID: "1e7b9a88-abe7-4f12-b2de-275d504fc326"). InnerVolumeSpecName "kube-api-access-gxtsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:24:04.977864 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.977830    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1e7b9a88-abe7-4f12-b2de-275d504fc326" (UID: "1e7b9a88-abe7-4f12-b2de-275d504fc326"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:24:04.977910 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:04.977865    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e7b9a88-abe7-4f12-b2de-275d504fc326" (UID: "1e7b9a88-abe7-4f12-b2de-275d504fc326"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:24:05.077119 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.077022    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:05.077119 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.077053    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxtsw\" (UniqueName: \"kubernetes.io/projected/1e7b9a88-abe7-4f12-b2de-275d504fc326-kube-api-access-gxtsw\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:05.077119 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.077063    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e7b9a88-abe7-4f12-b2de-275d504fc326-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:05.077119 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.077072    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwcv2\" (UniqueName: \"kubernetes.io/projected/2af4018b-d13e-41ac-9f84-762464587829-kube-api-access-gwcv2\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:05.077119 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.077081    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e7b9a88-abe7-4f12-b2de-275d504fc326-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:05.077119 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.077088    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af4018b-d13e-41ac-9f84-762464587829-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:05.276272 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.276241    2560 generic.go:356] "Generic (PLEG): container finished" podID="2af4018b-d13e-41ac-9f84-762464587829" containerID="bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a" exitCode=0
Sep 17 14:24:05.276485 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.276284    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"
Sep 17 14:24:05.276485 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.276332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd" event={"ID":"2af4018b-d13e-41ac-9f84-762464587829","Type":"ContainerDied","Data":"bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a"}
Sep 17 14:24:05.276485 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.276415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd" event={"ID":"2af4018b-d13e-41ac-9f84-762464587829","Type":"ContainerDied","Data":"3550e740cdecc1e063f4ec606acd87ad14d6894ddcbef2e72dee05cbf3bcc47e"}
Sep 17 14:24:05.276485 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.276445    2560 scope.go:117] "RemoveContainer" containerID="bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a"
Sep 17 14:24:05.277890 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.277868    2560 generic.go:356] "Generic (PLEG): container finished" podID="1e7b9a88-abe7-4f12-b2de-275d504fc326" containerID="221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6" exitCode=0
Sep 17 14:24:05.277997 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.277902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l" event={"ID":"1e7b9a88-abe7-4f12-b2de-275d504fc326","Type":"ContainerDied","Data":"221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6"}
Sep 17 14:24:05.277997 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.277979    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l" event={"ID":"1e7b9a88-abe7-4f12-b2de-275d504fc326","Type":"ContainerDied","Data":"a80a18a759be19bce7d1799b06eb84012e447c05f7ef6fd12aa9209ec56f8efe"}
Sep 17 14:24:05.278099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.277905    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"
Sep 17 14:24:05.285794 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.285772    2560 scope.go:117] "RemoveContainer" containerID="bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a"
Sep 17 14:24:05.286158 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:24:05.286140    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a\": container with ID starting with bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a not found: ID does not exist" containerID="bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a"
Sep 17 14:24:05.286236 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.286166    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a"} err="failed to get container status \"bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a\": rpc error: code = NotFound desc = could not find container \"bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a\": container with ID starting with bbb0c11fd5faa77e39230901879defdd9cfa5b989cb61717f481a2c4fc420c4a not found: ID does not exist"
Sep 17 14:24:05.286236 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.286182    2560 scope.go:117] "RemoveContainer" containerID="221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6"
Sep 17 14:24:05.295300 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.295280    2560 scope.go:117] "RemoveContainer" containerID="221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6"
Sep 17 14:24:05.295626 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:24:05.295606    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6\": container with ID starting with 221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6 not found: ID does not exist" containerID="221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6"
Sep 17 14:24:05.295701 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.295639    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6"} err="failed to get container status \"221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6\": rpc error: code = NotFound desc = could not find container \"221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6\": container with ID starting with 221ff6b48ccb7a040e6873fc0d1b691dae551990b3e5fc22f795fb83641ddef6 not found: ID does not exist"
Sep 17 14:24:05.303245 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.303214    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"]
Sep 17 14:24:05.310178 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.309951    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-68183c-65dbc544fc-qtk9l"]
Sep 17 14:24:05.320135 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.320104    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"]
Sep 17 14:24:05.323769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:05.323742    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-68183c-8c86449-cd6kd"]
Sep 17 14:24:06.966912 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:06.966874    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e7b9a88-abe7-4f12-b2de-275d504fc326" path="/var/lib/kubelet/pods/1e7b9a88-abe7-4f12-b2de-275d504fc326/volumes"
Sep 17 14:24:06.967295 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:06.967244    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af4018b-d13e-41ac-9f84-762464587829" path="/var/lib/kubelet/pods/2af4018b-d13e-41ac-9f84-762464587829/volumes"
Sep 17 14:24:14.093082 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.093038    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l" podUID="c876472f-197b-4c1b-8ed8-e03047bf64ec" containerName="main" containerID="cri-o://e33960d86461b3ca7eaf8f5856c19cb4ed6ccee5f0af992694c1357f0052d32a" gracePeriod=45
Sep 17 14:24:14.327880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.327847    2560 generic.go:356] "Generic (PLEG): container finished" podID="c876472f-197b-4c1b-8ed8-e03047bf64ec" containerID="e33960d86461b3ca7eaf8f5856c19cb4ed6ccee5f0af992694c1357f0052d32a" exitCode=0
Sep 17 14:24:14.328068 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.327926    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l" event={"ID":"c876472f-197b-4c1b-8ed8-e03047bf64ec","Type":"ContainerDied","Data":"e33960d86461b3ca7eaf8f5856c19cb4ed6ccee5f0af992694c1357f0052d32a"}
Sep 17 14:24:14.358675 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.358606    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:24:14.478169 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.478132    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-dshm\") pod \"c876472f-197b-4c1b-8ed8-e03047bf64ec\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") "
Sep 17 14:24:14.478472 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.478213    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-model-cache\") pod \"c876472f-197b-4c1b-8ed8-e03047bf64ec\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") "
Sep 17 14:24:14.478472 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.478236    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pz5g\" (UniqueName: \"kubernetes.io/projected/c876472f-197b-4c1b-8ed8-e03047bf64ec-kube-api-access-8pz5g\") pod \"c876472f-197b-4c1b-8ed8-e03047bf64ec\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") "
Sep 17 14:24:14.478472 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.478260    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-tmp-dir\") pod \"c876472f-197b-4c1b-8ed8-e03047bf64ec\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") "
Sep 17 14:24:14.478472 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.478290    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c876472f-197b-4c1b-8ed8-e03047bf64ec-tls-certs\") pod \"c876472f-197b-4c1b-8ed8-e03047bf64ec\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") "
Sep 17 14:24:14.478472 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.478310    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-home\") pod \"c876472f-197b-4c1b-8ed8-e03047bf64ec\" (UID: \"c876472f-197b-4c1b-8ed8-e03047bf64ec\") "
Sep 17 14:24:14.478769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.478596    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "c876472f-197b-4c1b-8ed8-e03047bf64ec" (UID: "c876472f-197b-4c1b-8ed8-e03047bf64ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:24:14.478829 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.478812    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:14.478969 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.478872    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-home" (OuterVolumeSpecName: "home") pod "c876472f-197b-4c1b-8ed8-e03047bf64ec" (UID: "c876472f-197b-4c1b-8ed8-e03047bf64ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:24:14.478969 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.478931    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c876472f-197b-4c1b-8ed8-e03047bf64ec" (UID: "c876472f-197b-4c1b-8ed8-e03047bf64ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:24:14.480673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.480647    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-dshm" (OuterVolumeSpecName: "dshm") pod "c876472f-197b-4c1b-8ed8-e03047bf64ec" (UID: "c876472f-197b-4c1b-8ed8-e03047bf64ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:24:14.480764 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.480730    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c876472f-197b-4c1b-8ed8-e03047bf64ec-kube-api-access-8pz5g" (OuterVolumeSpecName: "kube-api-access-8pz5g") pod "c876472f-197b-4c1b-8ed8-e03047bf64ec" (UID: "c876472f-197b-4c1b-8ed8-e03047bf64ec"). InnerVolumeSpecName "kube-api-access-8pz5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:24:14.480764 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.480730    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c876472f-197b-4c1b-8ed8-e03047bf64ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c876472f-197b-4c1b-8ed8-e03047bf64ec" (UID: "c876472f-197b-4c1b-8ed8-e03047bf64ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:24:14.579893 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.579848    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pz5g\" (UniqueName: \"kubernetes.io/projected/c876472f-197b-4c1b-8ed8-e03047bf64ec-kube-api-access-8pz5g\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:14.579893 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.579888    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:14.580095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.579904    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c876472f-197b-4c1b-8ed8-e03047bf64ec-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:14.580095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.579917    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:14.580095 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:14.579925    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c876472f-197b-4c1b-8ed8-e03047bf64ec-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:24:15.335217 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:15.335184    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"
Sep 17 14:24:15.335217 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:15.335195    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l" event={"ID":"c876472f-197b-4c1b-8ed8-e03047bf64ec","Type":"ContainerDied","Data":"5c4f1d5ee841471726fc4331f3658e4e0f9039255799096e27fd161bef8bd729"}
Sep 17 14:24:15.335811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:15.335251    2560 scope.go:117] "RemoveContainer" containerID="e33960d86461b3ca7eaf8f5856c19cb4ed6ccee5f0af992694c1357f0052d32a"
Sep 17 14:24:15.353206 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:15.353171    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"]
Sep 17 14:24:15.357663 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:15.357628    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-68183c/explicit-model-68183c-kserve-8646c6c9ff-x2d5l"]
Sep 17 14:24:16.966000 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:16.965964    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c876472f-197b-4c1b-8ed8-e03047bf64ec" path="/var/lib/kubelet/pods/c876472f-197b-4c1b-8ed8-e03047bf64ec/volumes"
Sep 17 14:24:17.898892 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.898853    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"]
Sep 17 14:24:17.899493 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899476    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="844bf3cb-8aaf-41d8-9928-49709d8f0ad0" containerName="maas-api"
Sep 17 14:24:17.899544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899496    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="844bf3cb-8aaf-41d8-9928-49709d8f0ad0" containerName="maas-api"
Sep 17 14:24:17.899544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899509    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e7b9a88-abe7-4f12-b2de-275d504fc326" containerName="maas-api"
Sep 17 14:24:17.899544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899518    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e7b9a88-abe7-4f12-b2de-275d504fc326" containerName="maas-api"
Sep 17 14:24:17.899544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899535    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c876472f-197b-4c1b-8ed8-e03047bf64ec" containerName="main"
Sep 17 14:24:17.899670 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899545    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c876472f-197b-4c1b-8ed8-e03047bf64ec" containerName="main"
Sep 17 14:24:17.899670 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899562    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af4018b-d13e-41ac-9f84-762464587829" containerName="payload-processing"
Sep 17 14:24:17.899670 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899571    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af4018b-d13e-41ac-9f84-762464587829" containerName="payload-processing"
Sep 17 14:24:17.899670 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899661    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e7b9a88-abe7-4f12-b2de-275d504fc326" containerName="maas-api"
Sep 17 14:24:17.899790 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899675    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c876472f-197b-4c1b-8ed8-e03047bf64ec" containerName="main"
Sep 17 14:24:17.899790 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899687    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af4018b-d13e-41ac-9f84-762464587829" containerName="payload-processing"
Sep 17 14:24:17.899790 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.899697    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="844bf3cb-8aaf-41d8-9928-49709d8f0ad0" containerName="maas-api"
Sep 17 14:24:17.904539 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.904505    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:17.907089 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.907066    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c98839-serving-cert\""
Sep 17 14:24:17.912269 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.912235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:17.912433 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.912302    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"]
Sep 17 14:24:17.912433 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.912334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:17.912574 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.912429    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72d81f3d-b91a-4cac-8670-06e6d2c64876-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:17.912574 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:17.912459    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfwt8\" (UniqueName: \"kubernetes.io/projected/72d81f3d-b91a-4cac-8670-06e6d2c64876-kube-api-access-vfwt8\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.013818 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.013779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.014305 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.013858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.014305 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.013900    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72d81f3d-b91a-4cac-8670-06e6d2c64876-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.014305 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.013927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwt8\" (UniqueName: \"kubernetes.io/projected/72d81f3d-b91a-4cac-8670-06e6d2c64876-kube-api-access-vfwt8\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.014305 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:24:18.014036    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-serving-cert: secret "maas-api-e2e-shared-a-w2-c98839-serving-cert" not found
Sep 17 14:24:18.014305 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:24:18.014151    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-maas-api-tls podName:72d81f3d-b91a-4cac-8670-06e6d2c64876 nodeName:}" failed. No retries permitted until 2026-09-17 14:24:18.514129395 +0000 UTC m=+1156.105305160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-maas-api-tls") pod "maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7" (UID: "72d81f3d-b91a-4cac-8670-06e6d2c64876") : secret "maas-api-e2e-shared-a-w2-c98839-serving-cert" not found
Sep 17 14:24:18.014783 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.014714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72d81f3d-b91a-4cac-8670-06e6d2c64876-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.016550 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.016529    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.028118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.028084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwt8\" (UniqueName: \"kubernetes.io/projected/72d81f3d-b91a-4cac-8670-06e6d2c64876-kube-api-access-vfwt8\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.519091 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.519045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.521715 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.521682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.621392 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.621344    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"]
Sep 17 14:24:18.625026 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.625004    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:24:18.627410 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.627353    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c98839\""
Sep 17 14:24:18.627410 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.627406    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c98839-dockercfg-kt2hc\""
Sep 17 14:24:18.633399 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.633372    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"]
Sep 17 14:24:18.659846 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.659809    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"]
Sep 17 14:24:18.663377 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.663338    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:24:18.672979 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.672951    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"]
Sep 17 14:24:18.720878 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.720836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f52hq\" (UniqueName: \"kubernetes.io/projected/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-kube-api-access-f52hq\") pod \"payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d\" (UID: \"a339aeec-c248-4a1d-bcfe-e4ba36b2c099\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:24:18.720878 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.720880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k99s\" (UniqueName: \"kubernetes.io/projected/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-kube-api-access-2k99s\") pod \"payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq\" (UID: \"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:24:18.721112 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.720909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d\" (UID: \"a339aeec-c248-4a1d-bcfe-e4ba36b2c099\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:24:18.721112 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.721047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq\" (UID: \"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:24:18.818579 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.818492    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:18.822072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.821613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f52hq\" (UniqueName: \"kubernetes.io/projected/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-kube-api-access-f52hq\") pod \"payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d\" (UID: \"a339aeec-c248-4a1d-bcfe-e4ba36b2c099\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:24:18.822072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.821644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k99s\" (UniqueName: \"kubernetes.io/projected/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-kube-api-access-2k99s\") pod \"payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq\" (UID: \"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:24:18.822072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.821677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d\" (UID: \"a339aeec-c248-4a1d-bcfe-e4ba36b2c099\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:24:18.822072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.821797    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq\" (UID: \"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:24:18.822554 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.822531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d\" (UID: \"a339aeec-c248-4a1d-bcfe-e4ba36b2c099\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:24:18.822627 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.822572    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq\" (UID: \"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:24:18.830506 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.830477    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f52hq\" (UniqueName: \"kubernetes.io/projected/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-kube-api-access-f52hq\") pod \"payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d\" (UID: \"a339aeec-c248-4a1d-bcfe-e4ba36b2c099\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:24:18.830632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.830576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k99s\" (UniqueName: \"kubernetes.io/projected/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-kube-api-access-2k99s\") pod \"payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq\" (UID: \"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:24:18.938020 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.937979    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:24:18.975020 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:18.974990    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:24:19.098294 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.098209    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"]
Sep 17 14:24:19.101938 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:24:19.101901    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d81f3d_b91a_4cac_8670_06e6d2c64876.slice/crio-990a7b56b47268dac3137beb520427bb36dd2abb68337c5fd1ea5393a4a94d89 WatchSource:0}: Error finding container 990a7b56b47268dac3137beb520427bb36dd2abb68337c5fd1ea5393a4a94d89: Status 404 returned error can't find the container with id 990a7b56b47268dac3137beb520427bb36dd2abb68337c5fd1ea5393a4a94d89
Sep 17 14:24:19.219026 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.218999    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"]
Sep 17 14:24:19.221312 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:24:19.221280    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e56b1f_513e_4c07_a8d7_5d84406ae6d7.slice/crio-89a216fb33d7b073d883ccbbbce4630c109aa399ed9afaf915a6233e5c4222cd WatchSource:0}: Error finding container 89a216fb33d7b073d883ccbbbce4630c109aa399ed9afaf915a6233e5c4222cd: Status 404 returned error can't find the container with id 89a216fb33d7b073d883ccbbbce4630c109aa399ed9afaf915a6233e5c4222cd
Sep 17 14:24:19.268455 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.268277    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"]
Sep 17 14:24:19.270604 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:24:19.270578    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda339aeec_c248_4a1d_bcfe_e4ba36b2c099.slice/crio-03944ebab18bddea2b651e4b1ad111ec76f0dc2b15596591a29ce18554f5bfc4 WatchSource:0}: Error finding container 03944ebab18bddea2b651e4b1ad111ec76f0dc2b15596591a29ce18554f5bfc4: Status 404 returned error can't find the container with id 03944ebab18bddea2b651e4b1ad111ec76f0dc2b15596591a29ce18554f5bfc4
Sep 17 14:24:19.359585 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.359394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d" event={"ID":"a339aeec-c248-4a1d-bcfe-e4ba36b2c099","Type":"ContainerStarted","Data":"af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416"}
Sep 17 14:24:19.359585 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.359474    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:24:19.359585 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.359497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d" event={"ID":"a339aeec-c248-4a1d-bcfe-e4ba36b2c099","Type":"ContainerStarted","Data":"03944ebab18bddea2b651e4b1ad111ec76f0dc2b15596591a29ce18554f5bfc4"}
Sep 17 14:24:19.361090 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.361062    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7" event={"ID":"72d81f3d-b91a-4cac-8670-06e6d2c64876","Type":"ContainerStarted","Data":"990a7b56b47268dac3137beb520427bb36dd2abb68337c5fd1ea5393a4a94d89"}
Sep 17 14:24:19.363158 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.363126    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq" event={"ID":"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7","Type":"ContainerStarted","Data":"3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da"}
Sep 17 14:24:19.363273 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.363182    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq" event={"ID":"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7","Type":"ContainerStarted","Data":"89a216fb33d7b073d883ccbbbce4630c109aa399ed9afaf915a6233e5c4222cd"}
Sep 17 14:24:19.363273 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.363222    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:24:19.382099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.382033    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d" podStartSLOduration=1.382010233 podStartE2EDuration="1.382010233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:19.381369058 +0000 UTC m=+1156.972544814" watchObservedRunningTime="2026-09-17 14:24:19.382010233 +0000 UTC m=+1156.973185999"
Sep 17 14:24:19.409642 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:19.409541    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq" podStartSLOduration=1.409518883 podStartE2EDuration="1.409518883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:19.407880709 +0000 UTC m=+1156.999056472" watchObservedRunningTime="2026-09-17 14:24:19.409518883 +0000 UTC m=+1157.000694645"
Sep 17 14:24:20.370846 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:20.370799    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7" event={"ID":"72d81f3d-b91a-4cac-8670-06e6d2c64876","Type":"ContainerStarted","Data":"2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769"}
Sep 17 14:24:20.371348 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:20.371078    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:20.391235 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:20.391178    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7" podStartSLOduration=2.925250067 podStartE2EDuration="3.391160737s" totalImagesPullingTime="465.91067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:20.388848811 +0000 UTC m=+1157.980024574" watchObservedRunningTime="2026-09-17 14:24:20.391160737 +0000 UTC m=+1157.982336498"
Sep 17 14:24:26.383136 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:26.383061    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:24:30.373575 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:30.373539    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:24:30.374070 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:30.373636    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:24:48.201402 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.201365    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"]
Sep 17 14:24:48.205251 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.205227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.208235 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.208213    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1269b2\"/\"distinct-w0-1269b2-kserve-self-signed-certs\""
Sep 17 14:24:48.234367 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.232761    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"]
Sep 17 14:24:48.336876 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.336830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbd75\" (UniqueName: \"kubernetes.io/projected/1a8a49ee-08f1-49d0-9430-0d133f216b5e-kube-api-access-cbd75\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.336876 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.336878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-model-cache\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.337101 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.336978    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tmp-dir\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.337101 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.337029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-dshm\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.337101 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.337078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-home\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.337101 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.337097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tls-certs\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.437992 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.437950    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbd75\" (UniqueName: \"kubernetes.io/projected/1a8a49ee-08f1-49d0-9430-0d133f216b5e-kube-api-access-cbd75\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.437992 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.437995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-model-cache\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.438228 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.438146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tmp-dir\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.438283 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.438225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-dshm\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.438412 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.438286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-home\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.438412 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.438318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tls-certs\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.438545 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.438522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-model-cache\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.438602 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.438538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tmp-dir\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.438711 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.438689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-home\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.440668 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.440645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-dshm\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.440771 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.440761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tls-certs\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.446641 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.446617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbd75\" (UniqueName: \"kubernetes.io/projected/1a8a49ee-08f1-49d0-9430-0d133f216b5e-kube-api-access-cbd75\") pod \"distinct-w0-1269b2-kserve-6c9c54ff97-ljk96\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.514471 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.514443    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:48.794338 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:48.794312    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"]
Sep 17 14:24:48.796574 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:24:48.796546    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a8a49ee_08f1_49d0_9430_0d133f216b5e.slice/crio-5e05c7ff15913c5c8a0b557623d17bac167f8139bd01245a2b031a7a7a1059df WatchSource:0}: Error finding container 5e05c7ff15913c5c8a0b557623d17bac167f8139bd01245a2b031a7a7a1059df: Status 404 returned error can't find the container with id 5e05c7ff15913c5c8a0b557623d17bac167f8139bd01245a2b031a7a7a1059df
Sep 17 14:24:49.523445 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:49.523411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96" event={"ID":"1a8a49ee-08f1-49d0-9430-0d133f216b5e","Type":"ContainerStarted","Data":"a0266001fd8a8179a4805f9a975b2d1a81c50d0f2e9bc823f8152f37e4be4965"}
Sep 17 14:24:49.523847 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:49.523456    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96" event={"ID":"1a8a49ee-08f1-49d0-9430-0d133f216b5e","Type":"ContainerStarted","Data":"5e05c7ff15913c5c8a0b557623d17bac167f8139bd01245a2b031a7a7a1059df"}
Sep 17 14:24:52.706986 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.706928    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96" podStartSLOduration=4.706907295 podStartE2EDuration="4.706907295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:49.544918502 +0000 UTC m=+1187.136094265" watchObservedRunningTime="2026-09-17 14:24:52.706907295 +0000 UTC m=+1190.298083057"
Sep 17 14:24:52.707454 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.707185    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"]
Sep 17 14:24:52.711002 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.710984    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:24:52.714093 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.714061    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b83f70-dockercfg-fq7vx\""
Sep 17 14:24:52.714093 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.714066    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b83f70\""
Sep 17 14:24:52.726775 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.726750    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"]
Sep 17 14:24:52.782608 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.782572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdbw4\" (UniqueName: \"kubernetes.io/projected/c887c738-bb81-4abe-96df-7b2ce5bac1c8-kube-api-access-hdbw4\") pod \"payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg\" (UID: \"c887c738-bb81-4abe-96df-7b2ce5bac1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:24:52.782790 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.782623    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c887c738-bb81-4abe-96df-7b2ce5bac1c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg\" (UID: \"c887c738-bb81-4abe-96df-7b2ce5bac1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:24:52.883259 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.883219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdbw4\" (UniqueName: \"kubernetes.io/projected/c887c738-bb81-4abe-96df-7b2ce5bac1c8-kube-api-access-hdbw4\") pod \"payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg\" (UID: \"c887c738-bb81-4abe-96df-7b2ce5bac1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:24:52.883480 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.883271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c887c738-bb81-4abe-96df-7b2ce5bac1c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg\" (UID: \"c887c738-bb81-4abe-96df-7b2ce5bac1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:24:52.884020 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.884001    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c887c738-bb81-4abe-96df-7b2ce5bac1c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg\" (UID: \"c887c738-bb81-4abe-96df-7b2ce5bac1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:24:52.891518 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:52.891493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdbw4\" (UniqueName: \"kubernetes.io/projected/c887c738-bb81-4abe-96df-7b2ce5bac1c8-kube-api-access-hdbw4\") pod \"payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg\" (UID: \"c887c738-bb81-4abe-96df-7b2ce5bac1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:24:53.022116 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:53.022081    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:24:53.296705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:53.296617    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"]
Sep 17 14:24:53.299930 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:24:53.299906    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc887c738_bb81_4abe_96df_7b2ce5bac1c8.slice/crio-6901461cdf82415ce3720ce312f4af9aa20cbefdec914a30b423f897058baeb0 WatchSource:0}: Error finding container 6901461cdf82415ce3720ce312f4af9aa20cbefdec914a30b423f897058baeb0: Status 404 returned error can't find the container with id 6901461cdf82415ce3720ce312f4af9aa20cbefdec914a30b423f897058baeb0
Sep 17 14:24:53.546283 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:53.546243    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg" event={"ID":"c887c738-bb81-4abe-96df-7b2ce5bac1c8","Type":"ContainerStarted","Data":"06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55"}
Sep 17 14:24:53.546489 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:53.546309    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg" event={"ID":"c887c738-bb81-4abe-96df-7b2ce5bac1c8","Type":"ContainerStarted","Data":"6901461cdf82415ce3720ce312f4af9aa20cbefdec914a30b423f897058baeb0"}
Sep 17 14:24:53.546489 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:53.546339    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:24:53.567659 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:53.567556    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg" podStartSLOduration=1.567538074 podStartE2EDuration="1.567538074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:53.565284432 +0000 UTC m=+1191.156460198" watchObservedRunningTime="2026-09-17 14:24:53.567538074 +0000 UTC m=+1191.158713877"
Sep 17 14:24:58.515475 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:58.515433    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:58.515903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:58.515494    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:58.527862 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:58.527828    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:24:58.584410 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:24:58.584374    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:25:04.553961 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:04.553927    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:25:36.130011 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.129975    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"]
Sep 17 14:25:36.133605 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.133587    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.136014 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.135985    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1a095586-serving-cert\""
Sep 17 14:25:36.144659 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.144631    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"]
Sep 17 14:25:36.245112 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.245073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-openshift-service-ca\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.245321 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.245213    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-maas-api-tls\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.245321 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.245315    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-metrics-certs\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.245458 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.245337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29576\" (UniqueName: \"kubernetes.io/projected/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-kube-api-access-29576\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.346917 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.346864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-maas-api-tls\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.347128 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.346980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-metrics-certs\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.347128 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.347010    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29576\" (UniqueName: \"kubernetes.io/projected/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-kube-api-access-29576\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.347128 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:25:36.347022    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-serving-cert: secret "maas-api-e2e-mt-1a095586-serving-cert" not found
Sep 17 14:25:36.347128 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.347041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-openshift-service-ca\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.347128 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:25:36.347094    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-maas-api-tls podName:0c3dc9b9-1f65-4747-9e36-9d604c1b6631 nodeName:}" failed. No retries permitted until 2026-09-17 14:25:36.847078785 +0000 UTC m=+1234.438254539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-maas-api-tls") pod "maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm" (UID: "0c3dc9b9-1f65-4747-9e36-9d604c1b6631") : secret "maas-api-e2e-mt-1a095586-serving-cert" not found
Sep 17 14:25:36.347736 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.347713    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-openshift-service-ca\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.349528 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.349509    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-metrics-certs\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.358039 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.358014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29576\" (UniqueName: \"kubernetes.io/projected/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-kube-api-access-29576\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.743007 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.742972    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"]
Sep 17 14:25:36.746723 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.746701    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:25:36.749725 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.749699    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1a095586-dockercfg-jxzkp\""
Sep 17 14:25:36.749877 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.749840    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1a095586\""
Sep 17 14:25:36.760500 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.760471    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"]
Sep 17 14:25:36.851150 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.851105    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f87e86-2f6c-4e09-bde6-8d85be4d6538-plugins-config-volume\") pod \"payload-processing-e2e-mt-1a095586-8677c674bf-tphqs\" (UID: \"67f87e86-2f6c-4e09-bde6-8d85be4d6538\") " pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:25:36.851346 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.851304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxzm\" (UniqueName: \"kubernetes.io/projected/67f87e86-2f6c-4e09-bde6-8d85be4d6538-kube-api-access-nxxzm\") pod \"payload-processing-e2e-mt-1a095586-8677c674bf-tphqs\" (UID: \"67f87e86-2f6c-4e09-bde6-8d85be4d6538\") " pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:25:36.851431 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.851409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-maas-api-tls\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.853878 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.853856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-maas-api-tls\") pod \"maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:36.952783 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.952743    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f87e86-2f6c-4e09-bde6-8d85be4d6538-plugins-config-volume\") pod \"payload-processing-e2e-mt-1a095586-8677c674bf-tphqs\" (UID: \"67f87e86-2f6c-4e09-bde6-8d85be4d6538\") " pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:25:36.952990 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.952864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxzm\" (UniqueName: \"kubernetes.io/projected/67f87e86-2f6c-4e09-bde6-8d85be4d6538-kube-api-access-nxxzm\") pod \"payload-processing-e2e-mt-1a095586-8677c674bf-tphqs\" (UID: \"67f87e86-2f6c-4e09-bde6-8d85be4d6538\") " pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:25:36.953547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.953522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f87e86-2f6c-4e09-bde6-8d85be4d6538-plugins-config-volume\") pod \"payload-processing-e2e-mt-1a095586-8677c674bf-tphqs\" (UID: \"67f87e86-2f6c-4e09-bde6-8d85be4d6538\") " pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:25:36.963805 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:36.963774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxzm\" (UniqueName: \"kubernetes.io/projected/67f87e86-2f6c-4e09-bde6-8d85be4d6538-kube-api-access-nxxzm\") pod \"payload-processing-e2e-mt-1a095586-8677c674bf-tphqs\" (UID: \"67f87e86-2f6c-4e09-bde6-8d85be4d6538\") " pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:25:37.046206 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:37.046117    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:37.058439 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:37.058414    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:25:37.334594 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:37.334561    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"]
Sep 17 14:25:37.336466 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:25:37.336440    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3dc9b9_1f65_4747_9e36_9d604c1b6631.slice/crio-42b1c43e4c754cfbed72ffae5f3d29b090e8eb2030ea7554bc511d8358276131 WatchSource:0}: Error finding container 42b1c43e4c754cfbed72ffae5f3d29b090e8eb2030ea7554bc511d8358276131: Status 404 returned error can't find the container with id 42b1c43e4c754cfbed72ffae5f3d29b090e8eb2030ea7554bc511d8358276131
Sep 17 14:25:37.358208 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:37.358177    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"]
Sep 17 14:25:37.359962 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:25:37.359932    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f87e86_2f6c_4e09_bde6_8d85be4d6538.slice/crio-371d6904d46fb263873ca787e0f38e830e07b195668e3b19e86099b9509bddee WatchSource:0}: Error finding container 371d6904d46fb263873ca787e0f38e830e07b195668e3b19e86099b9509bddee: Status 404 returned error can't find the container with id 371d6904d46fb263873ca787e0f38e830e07b195668e3b19e86099b9509bddee
Sep 17 14:25:37.781679 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:37.781638    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs" event={"ID":"67f87e86-2f6c-4e09-bde6-8d85be4d6538","Type":"ContainerStarted","Data":"113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03"}
Sep 17 14:25:37.781907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:37.781713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs" event={"ID":"67f87e86-2f6c-4e09-bde6-8d85be4d6538","Type":"ContainerStarted","Data":"371d6904d46fb263873ca787e0f38e830e07b195668e3b19e86099b9509bddee"}
Sep 17 14:25:37.781907 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:37.781747    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:25:37.782864 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:37.782840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm" event={"ID":"0c3dc9b9-1f65-4747-9e36-9d604c1b6631","Type":"ContainerStarted","Data":"42b1c43e4c754cfbed72ffae5f3d29b090e8eb2030ea7554bc511d8358276131"}
Sep 17 14:25:37.800581 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:37.800524    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs" podStartSLOduration=1.800506229 podStartE2EDuration="1.800506229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:25:37.798509614 +0000 UTC m=+1235.389685376" watchObservedRunningTime="2026-09-17 14:25:37.800506229 +0000 UTC m=+1235.391681991"
Sep 17 14:25:38.789647 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:38.789596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm" event={"ID":"0c3dc9b9-1f65-4747-9e36-9d604c1b6631","Type":"ContainerStarted","Data":"5deacce35e9cc763299c0785a0ed19b07952be0955ec4ee38cb7222981160186"}
Sep 17 14:25:38.790141 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:38.789909    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:38.825715 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:38.825665    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm" podStartSLOduration=2.2752206360000002 podStartE2EDuration="2.825649477s" totalImagesPullingTime="550.428841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:25:38.823332883 +0000 UTC m=+1236.414508645" watchObservedRunningTime="2026-09-17 14:25:38.825649477 +0000 UTC m=+1236.416825293"
Sep 17 14:25:44.800341 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:44.800310    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:25:48.792340 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:25:48.792306    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:26:04.465712 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.465658    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"]
Sep 17 14:26:04.468148 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.465931    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6" podUID="a3334318-e29e-48ea-9c05-b36424bef375" containerName="maas-api" containerID="cri-o://a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323" gracePeriod=30
Sep 17 14:26:04.731668 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.731643    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:26:04.841554 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.841521    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-maas-api-tls\") pod \"a3334318-e29e-48ea-9c05-b36424bef375\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") "
Sep 17 14:26:04.841759 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.841578    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-metrics-certs\") pod \"a3334318-e29e-48ea-9c05-b36424bef375\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") "
Sep 17 14:26:04.841759 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.841715    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8hhg\" (UniqueName: \"kubernetes.io/projected/a3334318-e29e-48ea-9c05-b36424bef375-kube-api-access-m8hhg\") pod \"a3334318-e29e-48ea-9c05-b36424bef375\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") "
Sep 17 14:26:04.841759 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.841735    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3334318-e29e-48ea-9c05-b36424bef375-openshift-service-ca\") pod \"a3334318-e29e-48ea-9c05-b36424bef375\" (UID: \"a3334318-e29e-48ea-9c05-b36424bef375\") "
Sep 17 14:26:04.842384 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.842336    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3334318-e29e-48ea-9c05-b36424bef375-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3334318-e29e-48ea-9c05-b36424bef375" (UID: "a3334318-e29e-48ea-9c05-b36424bef375"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:26:04.843753 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.843731    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a3334318-e29e-48ea-9c05-b36424bef375" (UID: "a3334318-e29e-48ea-9c05-b36424bef375"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:26:04.844224 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.844197    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3334318-e29e-48ea-9c05-b36424bef375" (UID: "a3334318-e29e-48ea-9c05-b36424bef375"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:26:04.844224 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.844213    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3334318-e29e-48ea-9c05-b36424bef375-kube-api-access-m8hhg" (OuterVolumeSpecName: "kube-api-access-m8hhg") pod "a3334318-e29e-48ea-9c05-b36424bef375" (UID: "a3334318-e29e-48ea-9c05-b36424bef375"). InnerVolumeSpecName "kube-api-access-m8hhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:26:04.927109 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.927069    2560 generic.go:356] "Generic (PLEG): container finished" podID="a3334318-e29e-48ea-9c05-b36424bef375" containerID="a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323" exitCode=0
Sep 17 14:26:04.927109 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.927109    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"
Sep 17 14:26:04.927331 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.927158    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6" event={"ID":"a3334318-e29e-48ea-9c05-b36424bef375","Type":"ContainerDied","Data":"a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323"}
Sep 17 14:26:04.927331 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.927207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6" event={"ID":"a3334318-e29e-48ea-9c05-b36424bef375","Type":"ContainerDied","Data":"e62cd6ed32ac7e9aa6ae8470cb84871dff2eba8a5221b7bedf085263b72012d0"}
Sep 17 14:26:04.927331 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.927230    2560 scope.go:117] "RemoveContainer" containerID="a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323"
Sep 17 14:26:04.936397 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.936379    2560 scope.go:117] "RemoveContainer" containerID="a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323"
Sep 17 14:26:04.936694 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:26:04.936674    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323\": container with ID starting with a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323 not found: ID does not exist" containerID="a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323"
Sep 17 14:26:04.936740 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.936703    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323"} err="failed to get container status \"a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323\": rpc error: code = NotFound desc = could not find container \"a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323\": container with ID starting with a74b2d391a378143cd22944fd287fe3ae56db3c5fbbb92729c745fb541ff8323 not found: ID does not exist"
Sep 17 14:26:04.943677 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.943641    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hhg\" (UniqueName: \"kubernetes.io/projected/a3334318-e29e-48ea-9c05-b36424bef375-kube-api-access-m8hhg\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:26:04.943677 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.943679    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3334318-e29e-48ea-9c05-b36424bef375-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:26:04.943884 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.943693    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:26:04.943884 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.943705    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3334318-e29e-48ea-9c05-b36424bef375-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:26:04.949876 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.949841    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"]
Sep 17 14:26:04.954661 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.954636    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-181b16-5ff9dbcfc4-nbxz6"]
Sep 17 14:26:04.967809 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:04.967783    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3334318-e29e-48ea-9c05-b36424bef375" path="/var/lib/kubelet/pods/a3334318-e29e-48ea-9c05-b36424bef375/volumes"
Sep 17 14:26:43.085675 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.085642    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"]
Sep 17 14:26:43.086145 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.086052    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3334318-e29e-48ea-9c05-b36424bef375" containerName="maas-api"
Sep 17 14:26:43.086145 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.086064    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3334318-e29e-48ea-9c05-b36424bef375" containerName="maas-api"
Sep 17 14:26:43.086145 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.086144    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3334318-e29e-48ea-9c05-b36424bef375" containerName="maas-api"
Sep 17 14:26:43.089538 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.089515    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.091895 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.091869    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1269b2\"/\"distinct-2-w0-1269b2-kserve-self-signed-certs\""
Sep 17 14:26:43.098341 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.098312    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"]
Sep 17 14:26:43.142901 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.142873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tmp-dir\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.143081 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.142911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-home\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.143081 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.142956    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-model-cache\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.143081 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.142984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tls-certs\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.143081 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.143000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkg9c\" (UniqueName: \"kubernetes.io/projected/3fbc4647-73e7-409c-afab-e14d82c4aeb8-kube-api-access-dkg9c\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.143238 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.143126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-dshm\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.243700 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.243662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-model-cache\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.243892 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.243714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tls-certs\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.243892 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.243733    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkg9c\" (UniqueName: \"kubernetes.io/projected/3fbc4647-73e7-409c-afab-e14d82c4aeb8-kube-api-access-dkg9c\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.243892 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.243811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-dshm\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.243892 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.243862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tmp-dir\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.244138 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.243898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-home\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.244138 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.244099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-model-cache\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.244246 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.244223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tmp-dir\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.244325 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.244304    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-home\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.246168 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.246138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-dshm\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.246466 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.246445    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tls-certs\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.252269 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.252246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkg9c\" (UniqueName: \"kubernetes.io/projected/3fbc4647-73e7-409c-afab-e14d82c4aeb8-kube-api-access-dkg9c\") pod \"distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") " pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.399903 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.399806    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:43.673521 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:43.673497    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"]
Sep 17 14:26:43.675220 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:26:43.675189    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fbc4647_73e7_409c_afab_e14d82c4aeb8.slice/crio-af677fc132b1af4aac9d9b1db1808da4d57a665e05f4b8c2db2666aa74632cc3 WatchSource:0}: Error finding container af677fc132b1af4aac9d9b1db1808da4d57a665e05f4b8c2db2666aa74632cc3: Status 404 returned error can't find the container with id af677fc132b1af4aac9d9b1db1808da4d57a665e05f4b8c2db2666aa74632cc3
Sep 17 14:26:44.135772 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:44.135734    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q" event={"ID":"3fbc4647-73e7-409c-afab-e14d82c4aeb8","Type":"ContainerStarted","Data":"90e79203442a2b4c9f806023405e77999ed4cdab45c8da7c0b7ebaa60178c144"}
Sep 17 14:26:44.135772 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:44.135778    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q" event={"ID":"3fbc4647-73e7-409c-afab-e14d82c4aeb8","Type":"ContainerStarted","Data":"af677fc132b1af4aac9d9b1db1808da4d57a665e05f4b8c2db2666aa74632cc3"}
Sep 17 14:26:44.156599 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:44.156538    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q" podStartSLOduration=1.156520971 podStartE2EDuration="1.156520971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:26:44.155049586 +0000 UTC m=+1301.746225347" watchObservedRunningTime="2026-09-17 14:26:44.156520971 +0000 UTC m=+1301.747696794"
Sep 17 14:26:53.400072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:53.400033    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:53.400785 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:53.400176    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:53.412567 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:53.412534    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:26:54.197232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:26:54.197204    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:27:45.323395 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.323348    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"]
Sep 17 14:27:45.323865 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.323567    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm" podUID="0c3dc9b9-1f65-4747-9e36-9d604c1b6631" containerName="maas-api" containerID="cri-o://5deacce35e9cc763299c0785a0ed19b07952be0955ec4ee38cb7222981160186" gracePeriod=30
Sep 17 14:27:45.411513 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.411474    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"]
Sep 17 14:27:45.411718 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.411695    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs" podUID="67f87e86-2f6c-4e09-bde6-8d85be4d6538" containerName="payload-processing" containerID="cri-o://113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03" gracePeriod=30
Sep 17 14:27:45.468055 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.468020    2560 generic.go:356] "Generic (PLEG): container finished" podID="0c3dc9b9-1f65-4747-9e36-9d604c1b6631" containerID="5deacce35e9cc763299c0785a0ed19b07952be0955ec4ee38cb7222981160186" exitCode=0
Sep 17 14:27:45.468241 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.468085    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm" event={"ID":"0c3dc9b9-1f65-4747-9e36-9d604c1b6631","Type":"ContainerDied","Data":"5deacce35e9cc763299c0785a0ed19b07952be0955ec4ee38cb7222981160186"}
Sep 17 14:27:45.659919 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.659893    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:27:45.692118 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.692090    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:27:45.703688 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.703652    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-openshift-service-ca\") pod \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") "
Sep 17 14:27:45.703864 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.703772    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-maas-api-tls\") pod \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") "
Sep 17 14:27:45.703916 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.703906    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-metrics-certs\") pod \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") "
Sep 17 14:27:45.704002 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.703950    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29576\" (UniqueName: \"kubernetes.io/projected/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-kube-api-access-29576\") pod \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\" (UID: \"0c3dc9b9-1f65-4747-9e36-9d604c1b6631\") "
Sep 17 14:27:45.704112 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.704052    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c3dc9b9-1f65-4747-9e36-9d604c1b6631" (UID: "0c3dc9b9-1f65-4747-9e36-9d604c1b6631"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:27:45.704319 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.704300    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:27:45.706056 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.706019    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0c3dc9b9-1f65-4747-9e36-9d604c1b6631" (UID: "0c3dc9b9-1f65-4747-9e36-9d604c1b6631"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:27:45.706186 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.706085    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-kube-api-access-29576" (OuterVolumeSpecName: "kube-api-access-29576") pod "0c3dc9b9-1f65-4747-9e36-9d604c1b6631" (UID: "0c3dc9b9-1f65-4747-9e36-9d604c1b6631"). InnerVolumeSpecName "kube-api-access-29576". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:27:45.706689 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.706668    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c3dc9b9-1f65-4747-9e36-9d604c1b6631" (UID: "0c3dc9b9-1f65-4747-9e36-9d604c1b6631"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:27:45.804926 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.804885    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f87e86-2f6c-4e09-bde6-8d85be4d6538-plugins-config-volume\") pod \"67f87e86-2f6c-4e09-bde6-8d85be4d6538\" (UID: \"67f87e86-2f6c-4e09-bde6-8d85be4d6538\") "
Sep 17 14:27:45.804926 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.804930    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxxzm\" (UniqueName: \"kubernetes.io/projected/67f87e86-2f6c-4e09-bde6-8d85be4d6538-kube-api-access-nxxzm\") pod \"67f87e86-2f6c-4e09-bde6-8d85be4d6538\" (UID: \"67f87e86-2f6c-4e09-bde6-8d85be4d6538\") "
Sep 17 14:27:45.805250 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.805236    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:27:45.805299 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.805252    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29576\" (UniqueName: \"kubernetes.io/projected/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-kube-api-access-29576\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:27:45.805299 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.805258    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f87e86-2f6c-4e09-bde6-8d85be4d6538-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67f87e86-2f6c-4e09-bde6-8d85be4d6538" (UID: "67f87e86-2f6c-4e09-bde6-8d85be4d6538"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:27:45.805299 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.805266    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3dc9b9-1f65-4747-9e36-9d604c1b6631-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:27:45.807076 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.807054    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f87e86-2f6c-4e09-bde6-8d85be4d6538-kube-api-access-nxxzm" (OuterVolumeSpecName: "kube-api-access-nxxzm") pod "67f87e86-2f6c-4e09-bde6-8d85be4d6538" (UID: "67f87e86-2f6c-4e09-bde6-8d85be4d6538"). InnerVolumeSpecName "kube-api-access-nxxzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:27:45.906994 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.906904    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f87e86-2f6c-4e09-bde6-8d85be4d6538-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:27:45.906994 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:45.906934    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxxzm\" (UniqueName: \"kubernetes.io/projected/67f87e86-2f6c-4e09-bde6-8d85be4d6538-kube-api-access-nxxzm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:27:46.473980 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.473948    2560 generic.go:356] "Generic (PLEG): container finished" podID="67f87e86-2f6c-4e09-bde6-8d85be4d6538" containerID="113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03" exitCode=0
Sep 17 14:27:46.474471 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.473992    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"
Sep 17 14:27:46.474471 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.474033    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs" event={"ID":"67f87e86-2f6c-4e09-bde6-8d85be4d6538","Type":"ContainerDied","Data":"113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03"}
Sep 17 14:27:46.474471 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.474084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs" event={"ID":"67f87e86-2f6c-4e09-bde6-8d85be4d6538","Type":"ContainerDied","Data":"371d6904d46fb263873ca787e0f38e830e07b195668e3b19e86099b9509bddee"}
Sep 17 14:27:46.474471 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.474113    2560 scope.go:117] "RemoveContainer" containerID="113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03"
Sep 17 14:27:46.475584 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.475560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm" event={"ID":"0c3dc9b9-1f65-4747-9e36-9d604c1b6631","Type":"ContainerDied","Data":"42b1c43e4c754cfbed72ffae5f3d29b090e8eb2030ea7554bc511d8358276131"}
Sep 17 14:27:46.475672 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.475578    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"
Sep 17 14:27:46.483525 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.483504    2560 scope.go:117] "RemoveContainer" containerID="113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03"
Sep 17 14:27:46.483850 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:27:46.483830    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03\": container with ID starting with 113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03 not found: ID does not exist" containerID="113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03"
Sep 17 14:27:46.483931 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.483862    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03"} err="failed to get container status \"113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03\": rpc error: code = NotFound desc = could not find container \"113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03\": container with ID starting with 113e79c7483ec6c1da9b0149fc84cf8a85eb1e05ce831c7fd3e38b6fcddb6d03 not found: ID does not exist"
Sep 17 14:27:46.483931 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.483884    2560 scope.go:117] "RemoveContainer" containerID="5deacce35e9cc763299c0785a0ed19b07952be0955ec4ee38cb7222981160186"
Sep 17 14:27:46.504557 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.504527    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"]
Sep 17 14:27:46.509057 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.509029    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a095586-76f67bfb8d-2snmm"]
Sep 17 14:27:46.522198 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.522166    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"]
Sep 17 14:27:46.525967 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.525942    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a095586-8677c674bf-tphqs"]
Sep 17 14:27:46.966924 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.966888    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c3dc9b9-1f65-4747-9e36-9d604c1b6631" path="/var/lib/kubelet/pods/0c3dc9b9-1f65-4747-9e36-9d604c1b6631/volumes"
Sep 17 14:27:46.967262 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:27:46.967249    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f87e86-2f6c-4e09-bde6-8d85be4d6538" path="/var/lib/kubelet/pods/67f87e86-2f6c-4e09-bde6-8d85be4d6538/volumes"
Sep 17 14:29:38.390294 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.390256    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"]
Sep 17 14:29:38.390909 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.390693    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f87e86-2f6c-4e09-bde6-8d85be4d6538" containerName="payload-processing"
Sep 17 14:29:38.390909 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.390705    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f87e86-2f6c-4e09-bde6-8d85be4d6538" containerName="payload-processing"
Sep 17 14:29:38.390909 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.390716    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3dc9b9-1f65-4747-9e36-9d604c1b6631" containerName="maas-api"
Sep 17 14:29:38.390909 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.390722    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3dc9b9-1f65-4747-9e36-9d604c1b6631" containerName="maas-api"
Sep 17 14:29:38.390909 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.390802    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f87e86-2f6c-4e09-bde6-8d85be4d6538" containerName="payload-processing"
Sep 17 14:29:38.390909 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.390809    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c3dc9b9-1f65-4747-9e36-9d604c1b6631" containerName="maas-api"
Sep 17 14:29:38.393886 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.393868    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.397044 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.397020    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fcd9f3\"/\"distinct-2-w4-fcd9f3-kserve-self-signed-certs\""
Sep 17 14:29:38.404808 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.404781    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"]
Sep 17 14:29:38.484969 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.484933    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tmp-dir\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.485141 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.484982    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-dshm\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.485141 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.485065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5j9z\" (UniqueName: \"kubernetes.io/projected/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-kube-api-access-g5j9z\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.485141 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.485103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-home\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.485141 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.485131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tls-certs\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.485280 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.485187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-model-cache\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.585867 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.585835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-dshm\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.586077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.585889    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5j9z\" (UniqueName: \"kubernetes.io/projected/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-kube-api-access-g5j9z\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.586077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.585918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-home\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.586077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.585941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tls-certs\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.586077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.585986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-model-cache\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.586077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.586055    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tmp-dir\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.586372 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.586334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-home\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.586426 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.586408    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-model-cache\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.586466 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.586448    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tmp-dir\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.588129 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.588111    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-dshm\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.588448 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.588433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tls-certs\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.594094 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.594063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5j9z\" (UniqueName: \"kubernetes.io/projected/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-kube-api-access-g5j9z\") pod \"distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") " pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.703674 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.703571    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:38.976121 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:38.976092    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"]
Sep 17 14:29:38.978302 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:29:38.978258    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod865fe8b2_4ed4_44f5_9b87_d3ec41aeb1d8.slice/crio-53dd403bf576de45b66034ee7636989c36a55afa5b93e5b59c10b612fab1fdbd WatchSource:0}: Error finding container 53dd403bf576de45b66034ee7636989c36a55afa5b93e5b59c10b612fab1fdbd: Status 404 returned error can't find the container with id 53dd403bf576de45b66034ee7636989c36a55afa5b93e5b59c10b612fab1fdbd
Sep 17 14:29:39.050958 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:39.050922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk" event={"ID":"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8","Type":"ContainerStarted","Data":"087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9"}
Sep 17 14:29:39.051160 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:39.050965    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk" event={"ID":"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8","Type":"ContainerStarted","Data":"53dd403bf576de45b66034ee7636989c36a55afa5b93e5b59c10b612fab1fdbd"}
Sep 17 14:29:39.068400 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:39.068323    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk" podStartSLOduration=1.068304576 podStartE2EDuration="1.068304576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:39.067143997 +0000 UTC m=+1476.658319759" watchObservedRunningTime="2026-09-17 14:29:39.068304576 +0000 UTC m=+1476.659480339"
Sep 17 14:29:48.704071 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:48.704034    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:48.704474 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:48.704092    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:48.716151 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:48.716127    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:49.115972 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:49.115934    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:29:54.997795 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:54.997757    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"]
Sep 17 14:29:55.001632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.001605    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.004807 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.004313    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a954c8-serving-cert\""
Sep 17 14:29:55.013159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.013135    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"]
Sep 17 14:29:55.065684 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.065654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4mb\" (UniqueName: \"kubernetes.io/projected/14b6121a-8b7d-4c16-8573-77e1205a1ba4-kube-api-access-ml4mb\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.065869 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.065695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b6121a-8b7d-4c16-8573-77e1205a1ba4-openshift-service-ca\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.065948 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.065871    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-maas-api-tls\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.065948 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.065911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-metrics-certs\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.166792 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.166747    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-maas-api-tls\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.166792 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.166790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-metrics-certs\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.167024 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.166844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4mb\" (UniqueName: \"kubernetes.io/projected/14b6121a-8b7d-4c16-8573-77e1205a1ba4-kube-api-access-ml4mb\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.167024 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.166874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b6121a-8b7d-4c16-8573-77e1205a1ba4-openshift-service-ca\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.167024 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:29:55.166904    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-serving-cert: secret "maas-api-e2e-api-a-a954c8-serving-cert" not found
Sep 17 14:29:55.167024 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:29:55.166988    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-maas-api-tls podName:14b6121a-8b7d-4c16-8573-77e1205a1ba4 nodeName:}" failed. No retries permitted until 2026-09-17 14:29:55.666966485 +0000 UTC m=+1493.258142242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-maas-api-tls") pod "maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4" (UID: "14b6121a-8b7d-4c16-8573-77e1205a1ba4") : secret "maas-api-e2e-api-a-a954c8-serving-cert" not found
Sep 17 14:29:55.167708 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.167689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b6121a-8b7d-4c16-8573-77e1205a1ba4-openshift-service-ca\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.169232 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.169214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-metrics-certs\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.178209 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.178183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4mb\" (UniqueName: \"kubernetes.io/projected/14b6121a-8b7d-4c16-8573-77e1205a1ba4-kube-api-access-ml4mb\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.596984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.596942    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"]
Sep 17 14:29:55.600717 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.600698    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:29:55.603169 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.603148    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a954c8\""
Sep 17 14:29:55.610438 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.610415    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"]
Sep 17 14:29:55.673472 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.673435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmsg6\" (UniqueName: \"kubernetes.io/projected/d0b696cf-fed3-42da-9bc3-47be4a72ac56-kube-api-access-tmsg6\") pod \"payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s\" (UID: \"d0b696cf-fed3-42da-9bc3-47be4a72ac56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:29:55.673472 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.673482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-maas-api-tls\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.673697 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.673564    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b696cf-fed3-42da-9bc3-47be4a72ac56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s\" (UID: \"d0b696cf-fed3-42da-9bc3-47be4a72ac56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:29:55.675936 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.675917    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-maas-api-tls\") pod \"maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:55.774649 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.774615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b696cf-fed3-42da-9bc3-47be4a72ac56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s\" (UID: \"d0b696cf-fed3-42da-9bc3-47be4a72ac56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:29:55.774855 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.774773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmsg6\" (UniqueName: \"kubernetes.io/projected/d0b696cf-fed3-42da-9bc3-47be4a72ac56-kube-api-access-tmsg6\") pod \"payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s\" (UID: \"d0b696cf-fed3-42da-9bc3-47be4a72ac56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:29:55.775471 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.775447    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b696cf-fed3-42da-9bc3-47be4a72ac56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s\" (UID: \"d0b696cf-fed3-42da-9bc3-47be4a72ac56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:29:55.783519 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.783495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmsg6\" (UniqueName: \"kubernetes.io/projected/d0b696cf-fed3-42da-9bc3-47be4a72ac56-kube-api-access-tmsg6\") pod \"payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s\" (UID: \"d0b696cf-fed3-42da-9bc3-47be4a72ac56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:29:55.912225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.912124    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:29:55.915236 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:55.915211    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:56.187961 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:56.187937    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"]
Sep 17 14:29:56.189866 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:29:56.189841    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b696cf_fed3_42da_9bc3_47be4a72ac56.slice/crio-f08255ca3b40ad1ce9b9fe32e430a185d0d5f222bb7d5bc267eddd1840e981ad WatchSource:0}: Error finding container f08255ca3b40ad1ce9b9fe32e430a185d0d5f222bb7d5bc267eddd1840e981ad: Status 404 returned error can't find the container with id f08255ca3b40ad1ce9b9fe32e430a185d0d5f222bb7d5bc267eddd1840e981ad
Sep 17 14:29:56.206014 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:56.205991    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"]
Sep 17 14:29:56.208816 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:29:56.208794    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b6121a_8b7d_4c16_8573_77e1205a1ba4.slice/crio-bacf2aed21cfad2629a53f33deca19cae3d358e05b0f1943c868b968d9314a13 WatchSource:0}: Error finding container bacf2aed21cfad2629a53f33deca19cae3d358e05b0f1943c868b968d9314a13: Status 404 returned error can't find the container with id bacf2aed21cfad2629a53f33deca19cae3d358e05b0f1943c868b968d9314a13
Sep 17 14:29:56.210345 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:56.210325    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:29:57.149583 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:57.149542    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s" event={"ID":"d0b696cf-fed3-42da-9bc3-47be4a72ac56","Type":"ContainerStarted","Data":"bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3"}
Sep 17 14:29:57.149800 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:57.149611    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:29:57.149800 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:57.149635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s" event={"ID":"d0b696cf-fed3-42da-9bc3-47be4a72ac56","Type":"ContainerStarted","Data":"f08255ca3b40ad1ce9b9fe32e430a185d0d5f222bb7d5bc267eddd1840e981ad"}
Sep 17 14:29:57.151172 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:57.151146    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4" event={"ID":"14b6121a-8b7d-4c16-8573-77e1205a1ba4","Type":"ContainerStarted","Data":"5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d"}
Sep 17 14:29:57.151291 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:57.151188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4" event={"ID":"14b6121a-8b7d-4c16-8573-77e1205a1ba4","Type":"ContainerStarted","Data":"bacf2aed21cfad2629a53f33deca19cae3d358e05b0f1943c868b968d9314a13"}
Sep 17 14:29:57.151291 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:57.151277    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:29:57.169079 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:57.169030    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s" podStartSLOduration=2.169016014 podStartE2EDuration="2.169016014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:57.167323352 +0000 UTC m=+1494.758499117" watchObservedRunningTime="2026-09-17 14:29:57.169016014 +0000 UTC m=+1494.760191774"
Sep 17 14:29:57.187950 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:29:57.187894    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4" podStartSLOduration=2.701367428 podStartE2EDuration="3.18787289s" totalImagesPullingTime="486.505462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:57.185194148 +0000 UTC m=+1494.776369911" watchObservedRunningTime="2026-09-17 14:29:57.18787289 +0000 UTC m=+1494.779048653"
Sep 17 14:30:00.169799 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:00.169765    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn"]
Sep 17 14:30:00.173673 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:00.173654    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn"
Sep 17 14:30:00.178369 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:00.178328    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ddwrn\""
Sep 17 14:30:00.183620 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:00.183537    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn"]
Sep 17 14:30:00.187250 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:00.187224    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn"
Sep 17 14:30:00.464384 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:00.464344    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn"]
Sep 17 14:30:00.466878 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:30:00.466850    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9e38b8_54a5_4319_b1f3_d62f4503edae.slice/crio-8e08bb5602df503b3608b517afc9de5603348e3387e51812da9a5bd5c0c9d631 WatchSource:0}: Error finding container 8e08bb5602df503b3608b517afc9de5603348e3387e51812da9a5bd5c0c9d631: Status 404 returned error can't find the container with id 8e08bb5602df503b3608b517afc9de5603348e3387e51812da9a5bd5c0c9d631
Sep 17 14:30:01.174843 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:01.174808    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn" event={"ID":"eb9e38b8-54a5-4319-b1f3-d62f4503edae","Type":"ContainerStarted","Data":"8e08bb5602df503b3608b517afc9de5603348e3387e51812da9a5bd5c0c9d631"}
Sep 17 14:30:03.161618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:03.161589    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:30:03.187169 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:03.187129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn" event={"ID":"eb9e38b8-54a5-4319-b1f3-d62f4503edae","Type":"ContainerStarted","Data":"59293c186aa7f65cd0329dcbfcb6b63851d5dc9f5089305fcbdb01f6900d5cc2"}
Sep 17 14:30:04.193961 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:04.193867    2560 generic.go:356] "Generic (PLEG): container finished" podID="eb9e38b8-54a5-4319-b1f3-d62f4503edae" containerID="59293c186aa7f65cd0329dcbfcb6b63851d5dc9f5089305fcbdb01f6900d5cc2" exitCode=0
Sep 17 14:30:04.194315 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:04.193957    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn" event={"ID":"eb9e38b8-54a5-4319-b1f3-d62f4503edae","Type":"ContainerDied","Data":"59293c186aa7f65cd0329dcbfcb6b63851d5dc9f5089305fcbdb01f6900d5cc2"}
Sep 17 14:30:05.332527 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:05.332501    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn"
Sep 17 14:30:06.206207 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:06.206168    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn" event={"ID":"eb9e38b8-54a5-4319-b1f3-d62f4503edae","Type":"ContainerDied","Data":"8e08bb5602df503b3608b517afc9de5603348e3387e51812da9a5bd5c0c9d631"}
Sep 17 14:30:06.206207 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:06.206207    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn"
Sep 17 14:30:06.206450 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:06.206226    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e08bb5602df503b3608b517afc9de5603348e3387e51812da9a5bd5c0c9d631"
Sep 17 14:30:08.158302 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:08.158271    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:30:30.159098 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.159015    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"]
Sep 17 14:30:30.159880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.159507    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9e38b8-54a5-4319-b1f3-d62f4503edae" containerName="cleanup"
Sep 17 14:30:30.159880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.159521    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9e38b8-54a5-4319-b1f3-d62f4503edae" containerName="cleanup"
Sep 17 14:30:30.159880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.159626    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9e38b8-54a5-4319-b1f3-d62f4503edae" containerName="cleanup"
Sep 17 14:30:30.163948 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.163912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.166772 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.166751    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cf94fd-serving-cert\""
Sep 17 14:30:30.180920 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.180871    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"]
Sep 17 14:30:30.241740 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.241698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4hm\" (UniqueName: \"kubernetes.io/projected/f03cede8-2406-458f-906b-e58c41d0e39f-kube-api-access-mx4hm\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.241918 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.241759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03cede8-2406-458f-906b-e58c41d0e39f-openshift-service-ca\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.241918 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.241824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-metrics-certs\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.241918 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.241863    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-maas-api-tls\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.343194 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.343152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-maas-api-tls\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.343387 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.343274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4hm\" (UniqueName: \"kubernetes.io/projected/f03cede8-2406-458f-906b-e58c41d0e39f-kube-api-access-mx4hm\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.343387 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.343316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03cede8-2406-458f-906b-e58c41d0e39f-openshift-service-ca\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.343387 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:30:30.343316    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-serving-cert: secret "maas-api-e2e-api-b-cf94fd-serving-cert" not found
Sep 17 14:30:30.343387 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.343379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-metrics-certs\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.343560 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:30:30.343393    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-maas-api-tls podName:f03cede8-2406-458f-906b-e58c41d0e39f nodeName:}" failed. No retries permitted until 2026-09-17 14:30:30.843373144 +0000 UTC m=+1528.434548883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-maas-api-tls") pod "maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f" (UID: "f03cede8-2406-458f-906b-e58c41d0e39f") : secret "maas-api-e2e-api-b-cf94fd-serving-cert" not found
Sep 17 14:30:30.344054 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.344034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03cede8-2406-458f-906b-e58c41d0e39f-openshift-service-ca\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.345724 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.345708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-metrics-certs\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.357672 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.357643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4hm\" (UniqueName: \"kubernetes.io/projected/f03cede8-2406-458f-906b-e58c41d0e39f-kube-api-access-mx4hm\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.849252 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.849215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-maas-api-tls\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.851874 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.851838    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-maas-api-tls\") pod \"maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:30.877859 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.877827    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"]
Sep 17 14:30:30.881699 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.881680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:30:30.884304 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.884269    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cf94fd-dockercfg-5fvdm\""
Sep 17 14:30:30.884479 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.884461    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cf94fd\""
Sep 17 14:30:30.892618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:30.892593    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"]
Sep 17 14:30:31.052175 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:31.052128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d908727-b228-420f-9118-ff8ac0d56b71-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94\" (UID: \"1d908727-b228-420f-9118-ff8ac0d56b71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:30:31.052405 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:31.052226    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9kn4\" (UniqueName: \"kubernetes.io/projected/1d908727-b228-420f-9118-ff8ac0d56b71-kube-api-access-p9kn4\") pod \"payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94\" (UID: \"1d908727-b228-420f-9118-ff8ac0d56b71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:30:31.075169 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:31.075139    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:31.154015 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:31.153979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d908727-b228-420f-9118-ff8ac0d56b71-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94\" (UID: \"1d908727-b228-420f-9118-ff8ac0d56b71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:30:31.154214 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:31.154057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9kn4\" (UniqueName: \"kubernetes.io/projected/1d908727-b228-420f-9118-ff8ac0d56b71-kube-api-access-p9kn4\") pod \"payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94\" (UID: \"1d908727-b228-420f-9118-ff8ac0d56b71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:30:31.154868 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:31.154844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d908727-b228-420f-9118-ff8ac0d56b71-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94\" (UID: \"1d908727-b228-420f-9118-ff8ac0d56b71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:30:31.163851 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:31.163820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9kn4\" (UniqueName: \"kubernetes.io/projected/1d908727-b228-420f-9118-ff8ac0d56b71-kube-api-access-p9kn4\") pod \"payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94\" (UID: \"1d908727-b228-420f-9118-ff8ac0d56b71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:30:31.192429 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:31.192393    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:30:31.353059 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:31.353012    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"]
Sep 17 14:30:31.355211 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:30:31.355186    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03cede8_2406_458f_906b_e58c41d0e39f.slice/crio-e6445bf87923b110806c14e1c60459890a88b5c9c259d487d27467d01b45c77a WatchSource:0}: Error finding container e6445bf87923b110806c14e1c60459890a88b5c9c259d487d27467d01b45c77a: Status 404 returned error can't find the container with id e6445bf87923b110806c14e1c60459890a88b5c9c259d487d27467d01b45c77a
Sep 17 14:30:31.466797 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:31.466772    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"]
Sep 17 14:30:31.468089 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:30:31.468062    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d908727_b228_420f_9118_ff8ac0d56b71.slice/crio-7c8a1c446bf57a1105ec5240ce1550e93b7d0ae21fa258b26a64686cc448477a WatchSource:0}: Error finding container 7c8a1c446bf57a1105ec5240ce1550e93b7d0ae21fa258b26a64686cc448477a: Status 404 returned error can't find the container with id 7c8a1c446bf57a1105ec5240ce1550e93b7d0ae21fa258b26a64686cc448477a
Sep 17 14:30:32.354380 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:32.354334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94" event={"ID":"1d908727-b228-420f-9118-ff8ac0d56b71","Type":"ContainerStarted","Data":"2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89"}
Sep 17 14:30:32.354881 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:32.354423    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94" event={"ID":"1d908727-b228-420f-9118-ff8ac0d56b71","Type":"ContainerStarted","Data":"7c8a1c446bf57a1105ec5240ce1550e93b7d0ae21fa258b26a64686cc448477a"}
Sep 17 14:30:32.354881 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:32.354507    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:30:32.356010 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:32.355984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f" event={"ID":"f03cede8-2406-458f-906b-e58c41d0e39f","Type":"ContainerStarted","Data":"e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b"}
Sep 17 14:30:32.356130 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:32.356028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f" event={"ID":"f03cede8-2406-458f-906b-e58c41d0e39f","Type":"ContainerStarted","Data":"e6445bf87923b110806c14e1c60459890a88b5c9c259d487d27467d01b45c77a"}
Sep 17 14:30:32.356130 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:32.356077    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:32.373896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:32.373843    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94" podStartSLOduration=2.373826582 podStartE2EDuration="2.373826582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:32.372991862 +0000 UTC m=+1529.964167625" watchObservedRunningTime="2026-09-17 14:30:32.373826582 +0000 UTC m=+1529.965002343"
Sep 17 14:30:32.396459 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:32.396406    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f" podStartSLOduration=1.887784802 podStartE2EDuration="2.396388996s" totalImagesPullingTime="508.604194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:32.39444983 +0000 UTC m=+1529.985625595" watchObservedRunningTime="2026-09-17 14:30:32.396388996 +0000 UTC m=+1529.987564754"
Sep 17 14:30:38.367067 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:38.367032    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:30:43.364159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:30:43.364129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:31:13.604990 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:13.604952    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"]
Sep 17 14:31:13.607800 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:13.605204    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4" podUID="14b6121a-8b7d-4c16-8573-77e1205a1ba4" containerName="maas-api" containerID="cri-o://5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d" gracePeriod=30
Sep 17 14:31:13.667082 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:13.667045    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"]
Sep 17 14:31:13.667331 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:13.667307    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s" podUID="d0b696cf-fed3-42da-9bc3-47be4a72ac56" containerName="payload-pre-processing" containerID="cri-o://bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3" gracePeriod=30
Sep 17 14:31:13.953627 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:13.953600    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:31:13.956823 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:13.956795    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:31:14.021423 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.021392    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b696cf-fed3-42da-9bc3-47be4a72ac56-plugins-config-volume\") pod \"d0b696cf-fed3-42da-9bc3-47be4a72ac56\" (UID: \"d0b696cf-fed3-42da-9bc3-47be4a72ac56\") "
Sep 17 14:31:14.021621 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.021463    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-maas-api-tls\") pod \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") "
Sep 17 14:31:14.021621 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.021499    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml4mb\" (UniqueName: \"kubernetes.io/projected/14b6121a-8b7d-4c16-8573-77e1205a1ba4-kube-api-access-ml4mb\") pod \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") "
Sep 17 14:31:14.021621 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.021537    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b6121a-8b7d-4c16-8573-77e1205a1ba4-openshift-service-ca\") pod \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") "
Sep 17 14:31:14.021621 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.021610    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-metrics-certs\") pod \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\" (UID: \"14b6121a-8b7d-4c16-8573-77e1205a1ba4\") "
Sep 17 14:31:14.021847 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.021682    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmsg6\" (UniqueName: \"kubernetes.io/projected/d0b696cf-fed3-42da-9bc3-47be4a72ac56-kube-api-access-tmsg6\") pod \"d0b696cf-fed3-42da-9bc3-47be4a72ac56\" (UID: \"d0b696cf-fed3-42da-9bc3-47be4a72ac56\") "
Sep 17 14:31:14.021954 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.021768    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b696cf-fed3-42da-9bc3-47be4a72ac56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0b696cf-fed3-42da-9bc3-47be4a72ac56" (UID: "d0b696cf-fed3-42da-9bc3-47be4a72ac56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:14.022008 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.021938    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b6121a-8b7d-4c16-8573-77e1205a1ba4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14b6121a-8b7d-4c16-8573-77e1205a1ba4" (UID: "14b6121a-8b7d-4c16-8573-77e1205a1ba4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:14.022215 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.022160    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b696cf-fed3-42da-9bc3-47be4a72ac56-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:14.022215 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.022189    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b6121a-8b7d-4c16-8573-77e1205a1ba4-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:14.023711 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.023681    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b6121a-8b7d-4c16-8573-77e1205a1ba4-kube-api-access-ml4mb" (OuterVolumeSpecName: "kube-api-access-ml4mb") pod "14b6121a-8b7d-4c16-8573-77e1205a1ba4" (UID: "14b6121a-8b7d-4c16-8573-77e1205a1ba4"). InnerVolumeSpecName "kube-api-access-ml4mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:14.023809 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.023724    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14b6121a-8b7d-4c16-8573-77e1205a1ba4" (UID: "14b6121a-8b7d-4c16-8573-77e1205a1ba4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:31:14.024058 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.024027    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "14b6121a-8b7d-4c16-8573-77e1205a1ba4" (UID: "14b6121a-8b7d-4c16-8573-77e1205a1ba4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:31:14.024145 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.024038    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b696cf-fed3-42da-9bc3-47be4a72ac56-kube-api-access-tmsg6" (OuterVolumeSpecName: "kube-api-access-tmsg6") pod "d0b696cf-fed3-42da-9bc3-47be4a72ac56" (UID: "d0b696cf-fed3-42da-9bc3-47be4a72ac56"). InnerVolumeSpecName "kube-api-access-tmsg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:14.123795 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.123711    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:14.123795 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.123744    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmsg6\" (UniqueName: \"kubernetes.io/projected/d0b696cf-fed3-42da-9bc3-47be4a72ac56-kube-api-access-tmsg6\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:14.123795 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.123754    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b6121a-8b7d-4c16-8573-77e1205a1ba4-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:14.123795 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.123763    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml4mb\" (UniqueName: \"kubernetes.io/projected/14b6121a-8b7d-4c16-8573-77e1205a1ba4-kube-api-access-ml4mb\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:14.582883 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.582849    2560 generic.go:356] "Generic (PLEG): container finished" podID="d0b696cf-fed3-42da-9bc3-47be4a72ac56" containerID="bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3" exitCode=0
Sep 17 14:31:14.583070 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.582896    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"
Sep 17 14:31:14.583070 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.582939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s" event={"ID":"d0b696cf-fed3-42da-9bc3-47be4a72ac56","Type":"ContainerDied","Data":"bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3"}
Sep 17 14:31:14.583070 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.582993    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s" event={"ID":"d0b696cf-fed3-42da-9bc3-47be4a72ac56","Type":"ContainerDied","Data":"f08255ca3b40ad1ce9b9fe32e430a185d0d5f222bb7d5bc267eddd1840e981ad"}
Sep 17 14:31:14.583070 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.583021    2560 scope.go:117] "RemoveContainer" containerID="bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3"
Sep 17 14:31:14.584305 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.584283    2560 generic.go:356] "Generic (PLEG): container finished" podID="14b6121a-8b7d-4c16-8573-77e1205a1ba4" containerID="5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d" exitCode=0
Sep 17 14:31:14.584420 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.584323    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"
Sep 17 14:31:14.584420 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.584336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4" event={"ID":"14b6121a-8b7d-4c16-8573-77e1205a1ba4","Type":"ContainerDied","Data":"5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d"}
Sep 17 14:31:14.584420 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.584406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4" event={"ID":"14b6121a-8b7d-4c16-8573-77e1205a1ba4","Type":"ContainerDied","Data":"bacf2aed21cfad2629a53f33deca19cae3d358e05b0f1943c868b968d9314a13"}
Sep 17 14:31:14.593853 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.593831    2560 scope.go:117] "RemoveContainer" containerID="bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3"
Sep 17 14:31:14.594180 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:31:14.594157    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3\": container with ID starting with bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3 not found: ID does not exist" containerID="bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3"
Sep 17 14:31:14.594257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.594190    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3"} err="failed to get container status \"bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3\": rpc error: code = NotFound desc = could not find container \"bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3\": container with ID starting with bc74cc57fffd5c9335916232a2c75f418e0bfea2ad949b53e3d0211e953ea7f3 not found: ID does not exist"
Sep 17 14:31:14.594257 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.594209    2560 scope.go:117] "RemoveContainer" containerID="5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d"
Sep 17 14:31:14.602326 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.602310    2560 scope.go:117] "RemoveContainer" containerID="5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d"
Sep 17 14:31:14.602611 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:31:14.602594    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d\": container with ID starting with 5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d not found: ID does not exist" containerID="5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d"
Sep 17 14:31:14.602662 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.602619    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d"} err="failed to get container status \"5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d\": rpc error: code = NotFound desc = could not find container \"5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d\": container with ID starting with 5cb22fbf6f9b87023b90f60e1fa00840c70e6398e0d969db14d0ae752673396d not found: ID does not exist"
Sep 17 14:31:14.606599 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.606575    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"]
Sep 17 14:31:14.614825 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.614793    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a954c8-746bbd8478-h772s"]
Sep 17 14:31:14.624262 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.624240    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"]
Sep 17 14:31:14.628255 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.628232    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a954c8-75dff8d9d-hxpn4"]
Sep 17 14:31:14.966054 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.965972    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14b6121a-8b7d-4c16-8573-77e1205a1ba4" path="/var/lib/kubelet/pods/14b6121a-8b7d-4c16-8573-77e1205a1ba4/volumes"
Sep 17 14:31:14.966396 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:14.966353    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0b696cf-fed3-42da-9bc3-47be4a72ac56" path="/var/lib/kubelet/pods/d0b696cf-fed3-42da-9bc3-47be4a72ac56/volumes"
Sep 17 14:31:36.684219 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:36.684182    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"]
Sep 17 14:31:36.684784 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:36.684478    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f" podUID="f03cede8-2406-458f-906b-e58c41d0e39f" containerName="maas-api" containerID="cri-o://e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b" gracePeriod=30
Sep 17 14:31:36.741495 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:36.741460    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"]
Sep 17 14:31:36.741720 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:36.741697    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94" podUID="1d908727-b228-420f-9118-ff8ac0d56b71" containerName="payload-pre-processing" containerID="cri-o://2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89" gracePeriod=30
Sep 17 14:31:36.996826 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:36.996796    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:31:37.025730 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.025697    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:31:37.064742 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.064702    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-metrics-certs\") pod \"f03cede8-2406-458f-906b-e58c41d0e39f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") "
Sep 17 14:31:37.064922 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.064766    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9kn4\" (UniqueName: \"kubernetes.io/projected/1d908727-b228-420f-9118-ff8ac0d56b71-kube-api-access-p9kn4\") pod \"1d908727-b228-420f-9118-ff8ac0d56b71\" (UID: \"1d908727-b228-420f-9118-ff8ac0d56b71\") "
Sep 17 14:31:37.064922 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.064786    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d908727-b228-420f-9118-ff8ac0d56b71-plugins-config-volume\") pod \"1d908727-b228-420f-9118-ff8ac0d56b71\" (UID: \"1d908727-b228-420f-9118-ff8ac0d56b71\") "
Sep 17 14:31:37.065008 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.064959    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-maas-api-tls\") pod \"f03cede8-2406-458f-906b-e58c41d0e39f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") "
Sep 17 14:31:37.065008 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.065003    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx4hm\" (UniqueName: \"kubernetes.io/projected/f03cede8-2406-458f-906b-e58c41d0e39f-kube-api-access-mx4hm\") pod \"f03cede8-2406-458f-906b-e58c41d0e39f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") "
Sep 17 14:31:37.065119 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.065035    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03cede8-2406-458f-906b-e58c41d0e39f-openshift-service-ca\") pod \"f03cede8-2406-458f-906b-e58c41d0e39f\" (UID: \"f03cede8-2406-458f-906b-e58c41d0e39f\") "
Sep 17 14:31:37.065203 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.065170    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d908727-b228-420f-9118-ff8ac0d56b71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d908727-b228-420f-9118-ff8ac0d56b71" (UID: "1d908727-b228-420f-9118-ff8ac0d56b71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:37.065565 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.065529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03cede8-2406-458f-906b-e58c41d0e39f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f03cede8-2406-458f-906b-e58c41d0e39f" (UID: "f03cede8-2406-458f-906b-e58c41d0e39f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:37.065849 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.065828    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d908727-b228-420f-9118-ff8ac0d56b71-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:37.065924 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.065854    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03cede8-2406-458f-906b-e58c41d0e39f-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:37.067472 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.067443    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f03cede8-2406-458f-906b-e58c41d0e39f" (UID: "f03cede8-2406-458f-906b-e58c41d0e39f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:31:37.067599 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.067527    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f03cede8-2406-458f-906b-e58c41d0e39f" (UID: "f03cede8-2406-458f-906b-e58c41d0e39f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:31:37.067897 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.067875    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03cede8-2406-458f-906b-e58c41d0e39f-kube-api-access-mx4hm" (OuterVolumeSpecName: "kube-api-access-mx4hm") pod "f03cede8-2406-458f-906b-e58c41d0e39f" (UID: "f03cede8-2406-458f-906b-e58c41d0e39f"). InnerVolumeSpecName "kube-api-access-mx4hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:37.067959 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.067918    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d908727-b228-420f-9118-ff8ac0d56b71-kube-api-access-p9kn4" (OuterVolumeSpecName: "kube-api-access-p9kn4") pod "1d908727-b228-420f-9118-ff8ac0d56b71" (UID: "1d908727-b228-420f-9118-ff8ac0d56b71"). InnerVolumeSpecName "kube-api-access-p9kn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:37.166888 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.166855    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:37.166888 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.166884    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4hm\" (UniqueName: \"kubernetes.io/projected/f03cede8-2406-458f-906b-e58c41d0e39f-kube-api-access-mx4hm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:37.166888 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.166893    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f03cede8-2406-458f-906b-e58c41d0e39f-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:37.167116 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.166902    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9kn4\" (UniqueName: \"kubernetes.io/projected/1d908727-b228-420f-9118-ff8ac0d56b71-kube-api-access-p9kn4\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:37.711335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.711297    2560 generic.go:356] "Generic (PLEG): container finished" podID="1d908727-b228-420f-9118-ff8ac0d56b71" containerID="2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89" exitCode=0
Sep 17 14:31:37.711849 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.711350    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"
Sep 17 14:31:37.711849 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.711392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94" event={"ID":"1d908727-b228-420f-9118-ff8ac0d56b71","Type":"ContainerDied","Data":"2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89"}
Sep 17 14:31:37.711849 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.711446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94" event={"ID":"1d908727-b228-420f-9118-ff8ac0d56b71","Type":"ContainerDied","Data":"7c8a1c446bf57a1105ec5240ce1550e93b7d0ae21fa258b26a64686cc448477a"}
Sep 17 14:31:37.711849 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.711468    2560 scope.go:117] "RemoveContainer" containerID="2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89"
Sep 17 14:31:37.713040 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.712986    2560 generic.go:356] "Generic (PLEG): container finished" podID="f03cede8-2406-458f-906b-e58c41d0e39f" containerID="e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b" exitCode=0
Sep 17 14:31:37.713112 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.713046    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"
Sep 17 14:31:37.713112 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.713066    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f" event={"ID":"f03cede8-2406-458f-906b-e58c41d0e39f","Type":"ContainerDied","Data":"e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b"}
Sep 17 14:31:37.713224 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.713131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f" event={"ID":"f03cede8-2406-458f-906b-e58c41d0e39f","Type":"ContainerDied","Data":"e6445bf87923b110806c14e1c60459890a88b5c9c259d487d27467d01b45c77a"}
Sep 17 14:31:37.722430 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.722408    2560 scope.go:117] "RemoveContainer" containerID="2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89"
Sep 17 14:31:37.722767 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:31:37.722738    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89\": container with ID starting with 2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89 not found: ID does not exist" containerID="2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89"
Sep 17 14:31:37.722891 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.722779    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89"} err="failed to get container status \"2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89\": rpc error: code = NotFound desc = could not find container \"2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89\": container with ID starting with 2244fdf4d1a8678df6d49df56e1eeaeccb7e7b9a5fd35d4baf24c6cc587f2e89 not found: ID does not exist"
Sep 17 14:31:37.722891 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.722808    2560 scope.go:117] "RemoveContainer" containerID="e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b"
Sep 17 14:31:37.733164 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.733148    2560 scope.go:117] "RemoveContainer" containerID="e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b"
Sep 17 14:31:37.733469 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:31:37.733453    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b\": container with ID starting with e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b not found: ID does not exist" containerID="e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b"
Sep 17 14:31:37.733517 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.733476    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b"} err="failed to get container status \"e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b\": rpc error: code = NotFound desc = could not find container \"e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b\": container with ID starting with e62b827bc2a0e338c06bf4bbdbcc811ed4534cd0cc903cd365883cef233d493b not found: ID does not exist"
Sep 17 14:31:37.743434 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.743393    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"]
Sep 17 14:31:37.750968 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.750945    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf94fd-7d48ccfbdf-rxb7f"]
Sep 17 14:31:37.760553 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.760531    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"]
Sep 17 14:31:37.764076 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:37.764054    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf94fd-7bd4cc67d-h5w94"]
Sep 17 14:31:38.966199 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:38.966167    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d908727-b228-420f-9118-ff8ac0d56b71" path="/var/lib/kubelet/pods/1d908727-b228-420f-9118-ff8ac0d56b71/volumes"
Sep 17 14:31:38.966612 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:38.966601    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03cede8-2406-458f-906b-e58c41d0e39f" path="/var/lib/kubelet/pods/f03cede8-2406-458f-906b-e58c41d0e39f/volumes"
Sep 17 14:31:45.493620 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:45.493578    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"]
Sep 17 14:31:45.499498 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:45.499468    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"]
Sep 17 14:31:45.518317 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:45.518276    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"]
Sep 17 14:31:45.530651 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:45.530612    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"]
Sep 17 14:31:54.995231 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:54.995138    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"]
Sep 17 14:31:54.995712 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:54.995440    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7" podUID="72d81f3d-b91a-4cac-8670-06e6d2c64876" containerName="maas-api" containerID="cri-o://2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769" gracePeriod=30
Sep 17 14:31:55.027521 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.027483    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn"]
Sep 17 14:31:55.030410 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.030382    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c98839-29827590-bqzzn"]
Sep 17 14:31:55.055807 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.055768    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"]
Sep 17 14:31:55.056056 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.056017    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d" podUID="a339aeec-c248-4a1d-bcfe-e4ba36b2c099" containerName="payload-processing" containerID="cri-o://af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416" gracePeriod=30
Sep 17 14:31:55.062802 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.062768    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"]
Sep 17 14:31:55.063294 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.063263    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq" podUID="c0e56b1f-513e-4c07-a8d7-5d84406ae6d7" containerName="payload-pre-processing" containerID="cri-o://3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da" gracePeriod=30
Sep 17 14:31:55.369848 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.369813    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:31:55.371928 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.371907    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:31:55.373975 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.373952    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:31:55.459817 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.459782    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-plugins-config-volume\") pod \"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7\" (UID: \"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7\") "
Sep 17 14:31:55.460017 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.459894    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k99s\" (UniqueName: \"kubernetes.io/projected/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-kube-api-access-2k99s\") pod \"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7\" (UID: \"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7\") "
Sep 17 14:31:55.460017 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.459922    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72d81f3d-b91a-4cac-8670-06e6d2c64876-openshift-service-ca\") pod \"72d81f3d-b91a-4cac-8670-06e6d2c64876\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") "
Sep 17 14:31:55.460017 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.459978    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-metrics-certs\") pod \"72d81f3d-b91a-4cac-8670-06e6d2c64876\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") "
Sep 17 14:31:55.460193 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.460027    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-plugins-config-volume\") pod \"a339aeec-c248-4a1d-bcfe-e4ba36b2c099\" (UID: \"a339aeec-c248-4a1d-bcfe-e4ba36b2c099\") "
Sep 17 14:31:55.460193 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.460054    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfwt8\" (UniqueName: \"kubernetes.io/projected/72d81f3d-b91a-4cac-8670-06e6d2c64876-kube-api-access-vfwt8\") pod \"72d81f3d-b91a-4cac-8670-06e6d2c64876\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") "
Sep 17 14:31:55.460193 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.460086    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f52hq\" (UniqueName: \"kubernetes.io/projected/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-kube-api-access-f52hq\") pod \"a339aeec-c248-4a1d-bcfe-e4ba36b2c099\" (UID: \"a339aeec-c248-4a1d-bcfe-e4ba36b2c099\") "
Sep 17 14:31:55.460193 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.460151    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-maas-api-tls\") pod \"72d81f3d-b91a-4cac-8670-06e6d2c64876\" (UID: \"72d81f3d-b91a-4cac-8670-06e6d2c64876\") "
Sep 17 14:31:55.460399 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.460246    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0e56b1f-513e-4c07-a8d7-5d84406ae6d7" (UID: "c0e56b1f-513e-4c07-a8d7-5d84406ae6d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:55.460908 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.460541    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:55.460908 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.460743    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d81f3d-b91a-4cac-8670-06e6d2c64876-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72d81f3d-b91a-4cac-8670-06e6d2c64876" (UID: "72d81f3d-b91a-4cac-8670-06e6d2c64876"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:55.460908 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.460866    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a339aeec-c248-4a1d-bcfe-e4ba36b2c099" (UID: "a339aeec-c248-4a1d-bcfe-e4ba36b2c099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:55.462840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.462806    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "72d81f3d-b91a-4cac-8670-06e6d2c64876" (UID: "72d81f3d-b91a-4cac-8670-06e6d2c64876"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:31:55.462840 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.462815    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72d81f3d-b91a-4cac-8670-06e6d2c64876" (UID: "72d81f3d-b91a-4cac-8670-06e6d2c64876"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:31:55.463009 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.462872    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-kube-api-access-2k99s" (OuterVolumeSpecName: "kube-api-access-2k99s") pod "c0e56b1f-513e-4c07-a8d7-5d84406ae6d7" (UID: "c0e56b1f-513e-4c07-a8d7-5d84406ae6d7"). InnerVolumeSpecName "kube-api-access-2k99s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:55.463009 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.462989    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d81f3d-b91a-4cac-8670-06e6d2c64876-kube-api-access-vfwt8" (OuterVolumeSpecName: "kube-api-access-vfwt8") pod "72d81f3d-b91a-4cac-8670-06e6d2c64876" (UID: "72d81f3d-b91a-4cac-8670-06e6d2c64876"). InnerVolumeSpecName "kube-api-access-vfwt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:55.463115 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.463090    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-kube-api-access-f52hq" (OuterVolumeSpecName: "kube-api-access-f52hq") pod "a339aeec-c248-4a1d-bcfe-e4ba36b2c099" (UID: "a339aeec-c248-4a1d-bcfe-e4ba36b2c099"). InnerVolumeSpecName "kube-api-access-f52hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:55.562197 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.562106    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k99s\" (UniqueName: \"kubernetes.io/projected/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7-kube-api-access-2k99s\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:55.562197 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.562140    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72d81f3d-b91a-4cac-8670-06e6d2c64876-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:55.562197 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.562152    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:55.562197 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.562162    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:55.562197 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.562174    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfwt8\" (UniqueName: \"kubernetes.io/projected/72d81f3d-b91a-4cac-8670-06e6d2c64876-kube-api-access-vfwt8\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:55.562197 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.562182    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f52hq\" (UniqueName: \"kubernetes.io/projected/a339aeec-c248-4a1d-bcfe-e4ba36b2c099-kube-api-access-f52hq\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:55.562197 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.562190    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d81f3d-b91a-4cac-8670-06e6d2c64876-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:55.815885 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.815783    2560 generic.go:356] "Generic (PLEG): container finished" podID="72d81f3d-b91a-4cac-8670-06e6d2c64876" containerID="2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769" exitCode=0
Sep 17 14:31:55.815885 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.815832    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"
Sep 17 14:31:55.815885 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.815870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7" event={"ID":"72d81f3d-b91a-4cac-8670-06e6d2c64876","Type":"ContainerDied","Data":"2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769"}
Sep 17 14:31:55.816182 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.816003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7" event={"ID":"72d81f3d-b91a-4cac-8670-06e6d2c64876","Type":"ContainerDied","Data":"990a7b56b47268dac3137beb520427bb36dd2abb68337c5fd1ea5393a4a94d89"}
Sep 17 14:31:55.816182 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.816045    2560 scope.go:117] "RemoveContainer" containerID="2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769"
Sep 17 14:31:55.817422 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.817395    2560 generic.go:356] "Generic (PLEG): container finished" podID="c0e56b1f-513e-4c07-a8d7-5d84406ae6d7" containerID="3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da" exitCode=0
Sep 17 14:31:55.817596 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.817424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq" event={"ID":"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7","Type":"ContainerDied","Data":"3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da"}
Sep 17 14:31:55.817596 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.817449    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"
Sep 17 14:31:55.817596 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.817470    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq" event={"ID":"c0e56b1f-513e-4c07-a8d7-5d84406ae6d7","Type":"ContainerDied","Data":"89a216fb33d7b073d883ccbbbce4630c109aa399ed9afaf915a6233e5c4222cd"}
Sep 17 14:31:55.819555 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.819458    2560 generic.go:356] "Generic (PLEG): container finished" podID="a339aeec-c248-4a1d-bcfe-e4ba36b2c099" containerID="af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416" exitCode=0
Sep 17 14:31:55.819555 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.819492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d" event={"ID":"a339aeec-c248-4a1d-bcfe-e4ba36b2c099","Type":"ContainerDied","Data":"af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416"}
Sep 17 14:31:55.819555 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.819509    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"
Sep 17 14:31:55.819555 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.819543    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d" event={"ID":"a339aeec-c248-4a1d-bcfe-e4ba36b2c099","Type":"ContainerDied","Data":"03944ebab18bddea2b651e4b1ad111ec76f0dc2b15596591a29ce18554f5bfc4"}
Sep 17 14:31:55.826294 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.826265    2560 scope.go:117] "RemoveContainer" containerID="2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769"
Sep 17 14:31:55.826827 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:31:55.826794    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769\": container with ID starting with 2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769 not found: ID does not exist" containerID="2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769"
Sep 17 14:31:55.826936 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.826839    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769"} err="failed to get container status \"2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769\": rpc error: code = NotFound desc = could not find container \"2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769\": container with ID starting with 2ada3816f2459e87ca40cc0ef32330f57578b2272cb1019cffb73dd241b5d769 not found: ID does not exist"
Sep 17 14:31:55.826936 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.826863    2560 scope.go:117] "RemoveContainer" containerID="3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da"
Sep 17 14:31:55.837977 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.836486    2560 scope.go:117] "RemoveContainer" containerID="3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da"
Sep 17 14:31:55.837977 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:31:55.837914    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da\": container with ID starting with 3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da not found: ID does not exist" containerID="3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da"
Sep 17 14:31:55.837977 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.837951    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da"} err="failed to get container status \"3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da\": rpc error: code = NotFound desc = could not find container \"3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da\": container with ID starting with 3b202e30b75a532d4507aef9461b3623cfc34b1678ec91929a668120a02a95da not found: ID does not exist"
Sep 17 14:31:55.838154 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.837978    2560 scope.go:117] "RemoveContainer" containerID="af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416"
Sep 17 14:31:55.844472 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.844445    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"]
Sep 17 14:31:55.847543 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.847516    2560 scope.go:117] "RemoveContainer" containerID="af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416"
Sep 17 14:31:55.847923 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:31:55.847903    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416\": container with ID starting with af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416 not found: ID does not exist" containerID="af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416"
Sep 17 14:31:55.847981 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.847933    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416"} err="failed to get container status \"af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416\": rpc error: code = NotFound desc = could not find container \"af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416\": container with ID starting with af321a61e92cca4724ba93f1f9ae2c5362df6b458b77de756616087c442f7416 not found: ID does not exist"
Sep 17 14:31:55.849469 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.849450    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c98839-674d6646c8-2rpwq"]
Sep 17 14:31:55.859898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.859871    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"]
Sep 17 14:31:55.866799 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.866766    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c98839-6d5b54bc9c-d5h5d"]
Sep 17 14:31:55.885660 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.885530    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"]
Sep 17 14:31:55.886813 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:55.886780    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c98839-7f48fb5c99-pvbh7"]
Sep 17 14:31:56.392265 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.392231    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"]
Sep 17 14:31:56.392746 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.392534    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x" podUID="87254986-487e-4ac8-8c31-9ba553408fea" containerName="maas-api" containerID="cri-o://b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba" gracePeriod=30
Sep 17 14:31:56.463777 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.463741    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"]
Sep 17 14:31:56.464021 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.463990    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs" podUID="ad04203b-e2ad-4d57-bb53-46c43dfbc242" containerName="payload-pre-processing" containerID="cri-o://61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8" gracePeriod=30
Sep 17 14:31:56.727776 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.727747    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:31:56.737942 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.737913    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:31:56.830818 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.830779    2560 generic.go:356] "Generic (PLEG): container finished" podID="ad04203b-e2ad-4d57-bb53-46c43dfbc242" containerID="61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8" exitCode=0
Sep 17 14:31:56.830818 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.830818    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"
Sep 17 14:31:56.831115 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.830852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs" event={"ID":"ad04203b-e2ad-4d57-bb53-46c43dfbc242","Type":"ContainerDied","Data":"61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8"}
Sep 17 14:31:56.831115 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.830897    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs" event={"ID":"ad04203b-e2ad-4d57-bb53-46c43dfbc242","Type":"ContainerDied","Data":"25a54340af290ffbd7c395695393bd241204ca4426a5579a9d4269707f7c22bd"}
Sep 17 14:31:56.831115 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.830922    2560 scope.go:117] "RemoveContainer" containerID="61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8"
Sep 17 14:31:56.832212 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.832192    2560 generic.go:356] "Generic (PLEG): container finished" podID="87254986-487e-4ac8-8c31-9ba553408fea" containerID="b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba" exitCode=0
Sep 17 14:31:56.832301 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.832225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x" event={"ID":"87254986-487e-4ac8-8c31-9ba553408fea","Type":"ContainerDied","Data":"b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba"}
Sep 17 14:31:56.832301 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.832233    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"
Sep 17 14:31:56.832301 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.832260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x" event={"ID":"87254986-487e-4ac8-8c31-9ba553408fea","Type":"ContainerDied","Data":"7bf0be010c735f191d13c645e7b4465f81ba9308058db962151d6cb0138b0521"}
Sep 17 14:31:56.840203 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.840178    2560 scope.go:117] "RemoveContainer" containerID="61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8"
Sep 17 14:31:56.840587 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:31:56.840563    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8\": container with ID starting with 61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8 not found: ID does not exist" containerID="61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8"
Sep 17 14:31:56.840667 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.840596    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8"} err="failed to get container status \"61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8\": rpc error: code = NotFound desc = could not find container \"61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8\": container with ID starting with 61ff719971852ceaf7fb8d433869c36932d7a94022015afe735e17fd6b6ee5b8 not found: ID does not exist"
Sep 17 14:31:56.840667 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.840613    2560 scope.go:117] "RemoveContainer" containerID="b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba"
Sep 17 14:31:56.850136 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.850113    2560 scope.go:117] "RemoveContainer" containerID="b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba"
Sep 17 14:31:56.850532 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:31:56.850512    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba\": container with ID starting with b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba not found: ID does not exist" containerID="b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba"
Sep 17 14:31:56.850589 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.850540    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba"} err="failed to get container status \"b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba\": rpc error: code = NotFound desc = could not find container \"b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba\": container with ID starting with b3f230b09fa5c9973df53ccb1dfb73f6c7251fb8bee7b488030ea6794bcc70ba not found: ID does not exist"
Sep 17 14:31:56.875711 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.875630    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87254986-487e-4ac8-8c31-9ba553408fea-openshift-service-ca\") pod \"87254986-487e-4ac8-8c31-9ba553408fea\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") "
Sep 17 14:31:56.875711 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.875713    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-maas-api-tls\") pod \"87254986-487e-4ac8-8c31-9ba553408fea\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") "
Sep 17 14:31:56.875991 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.875768    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-metrics-certs\") pod \"87254986-487e-4ac8-8c31-9ba553408fea\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") "
Sep 17 14:31:56.875991 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.875829    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8msv\" (UniqueName: \"kubernetes.io/projected/87254986-487e-4ac8-8c31-9ba553408fea-kube-api-access-g8msv\") pod \"87254986-487e-4ac8-8c31-9ba553408fea\" (UID: \"87254986-487e-4ac8-8c31-9ba553408fea\") "
Sep 17 14:31:56.875991 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.875857    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad04203b-e2ad-4d57-bb53-46c43dfbc242-plugins-config-volume\") pod \"ad04203b-e2ad-4d57-bb53-46c43dfbc242\" (UID: \"ad04203b-e2ad-4d57-bb53-46c43dfbc242\") "
Sep 17 14:31:56.876142 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.876057    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87254986-487e-4ac8-8c31-9ba553408fea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87254986-487e-4ac8-8c31-9ba553408fea" (UID: "87254986-487e-4ac8-8c31-9ba553408fea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:56.876251 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.876221    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhgx8\" (UniqueName: \"kubernetes.io/projected/ad04203b-e2ad-4d57-bb53-46c43dfbc242-kube-api-access-bhgx8\") pod \"ad04203b-e2ad-4d57-bb53-46c43dfbc242\" (UID: \"ad04203b-e2ad-4d57-bb53-46c43dfbc242\") "
Sep 17 14:31:56.876324 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.876276    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad04203b-e2ad-4d57-bb53-46c43dfbc242-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad04203b-e2ad-4d57-bb53-46c43dfbc242" (UID: "ad04203b-e2ad-4d57-bb53-46c43dfbc242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:56.876803 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.876767    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad04203b-e2ad-4d57-bb53-46c43dfbc242-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:56.876803 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.876796    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87254986-487e-4ac8-8c31-9ba553408fea-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:56.878187 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.878158    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87254986-487e-4ac8-8c31-9ba553408fea" (UID: "87254986-487e-4ac8-8c31-9ba553408fea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:31:56.878314 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.878194    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "87254986-487e-4ac8-8c31-9ba553408fea" (UID: "87254986-487e-4ac8-8c31-9ba553408fea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:31:56.878404 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.878378    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87254986-487e-4ac8-8c31-9ba553408fea-kube-api-access-g8msv" (OuterVolumeSpecName: "kube-api-access-g8msv") pod "87254986-487e-4ac8-8c31-9ba553408fea" (UID: "87254986-487e-4ac8-8c31-9ba553408fea"). InnerVolumeSpecName "kube-api-access-g8msv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:56.878773 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.878742    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad04203b-e2ad-4d57-bb53-46c43dfbc242-kube-api-access-bhgx8" (OuterVolumeSpecName: "kube-api-access-bhgx8") pod "ad04203b-e2ad-4d57-bb53-46c43dfbc242" (UID: "ad04203b-e2ad-4d57-bb53-46c43dfbc242"). InnerVolumeSpecName "kube-api-access-bhgx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:56.966707 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.966624    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72d81f3d-b91a-4cac-8670-06e6d2c64876" path="/var/lib/kubelet/pods/72d81f3d-b91a-4cac-8670-06e6d2c64876/volumes"
Sep 17 14:31:56.967030 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.967016    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a339aeec-c248-4a1d-bcfe-e4ba36b2c099" path="/var/lib/kubelet/pods/a339aeec-c248-4a1d-bcfe-e4ba36b2c099/volumes"
Sep 17 14:31:56.967336 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.967324    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0e56b1f-513e-4c07-a8d7-5d84406ae6d7" path="/var/lib/kubelet/pods/c0e56b1f-513e-4c07-a8d7-5d84406ae6d7/volumes"
Sep 17 14:31:56.967590 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.967578    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9e38b8-54a5-4319-b1f3-d62f4503edae" path="/var/lib/kubelet/pods/eb9e38b8-54a5-4319-b1f3-d62f4503edae/volumes"
Sep 17 14:31:56.977603 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.977577    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:56.977603 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.977601    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87254986-487e-4ac8-8c31-9ba553408fea-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:56.977765 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.977611    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8msv\" (UniqueName: \"kubernetes.io/projected/87254986-487e-4ac8-8c31-9ba553408fea-kube-api-access-g8msv\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:56.977765 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:56.977620    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhgx8\" (UniqueName: \"kubernetes.io/projected/ad04203b-e2ad-4d57-bb53-46c43dfbc242-kube-api-access-bhgx8\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:31:57.149314 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:57.149275    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"]
Sep 17 14:31:57.153772 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:57.153725    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1269b2-8455c6cfbb-62rzs"]
Sep 17 14:31:57.166087 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:57.166052    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"]
Sep 17 14:31:57.169962 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:57.169935    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1269b2-5bcdf68988-zzx2x"]
Sep 17 14:31:58.966418 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:58.966373    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87254986-487e-4ac8-8c31-9ba553408fea" path="/var/lib/kubelet/pods/87254986-487e-4ac8-8c31-9ba553408fea/volumes"
Sep 17 14:31:58.966962 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:31:58.966941    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad04203b-e2ad-4d57-bb53-46c43dfbc242" path="/var/lib/kubelet/pods/ad04203b-e2ad-4d57-bb53-46c43dfbc242/volumes"
Sep 17 14:32:00.514105 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.514067    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q" podUID="3fbc4647-73e7-409c-afab-e14d82c4aeb8" containerName="main" containerID="cri-o://90e79203442a2b4c9f806023405e77999ed4cdab45c8da7c0b7ebaa60178c144" gracePeriod=45
Sep 17 14:32:00.519305 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.519257    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96" podUID="1a8a49ee-08f1-49d0-9430-0d133f216b5e" containerName="main" containerID="cri-o://a0266001fd8a8179a4805f9a975b2d1a81c50d0f2e9bc823f8152f37e4be4965" gracePeriod=45
Sep 17 14:32:00.551808 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.551765    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq" podUID="4843a90f-8f8f-4a04-93a0-1bb9dd20c850" containerName="main" containerID="cri-o://ee08be6e04f1b982e6f16c3bdcd82cf1f74aa7bf7c9e75241282db6b5aa385fd" gracePeriod=45
Sep 17 14:32:00.778639 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.778524    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f" podUID="b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" containerName="main" containerID="cri-o://8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a" gracePeriod=45
Sep 17 14:32:00.860632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.860600    2560 generic.go:356] "Generic (PLEG): container finished" podID="1a8a49ee-08f1-49d0-9430-0d133f216b5e" containerID="a0266001fd8a8179a4805f9a975b2d1a81c50d0f2e9bc823f8152f37e4be4965" exitCode=0
Sep 17 14:32:00.860811 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.860673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96" event={"ID":"1a8a49ee-08f1-49d0-9430-0d133f216b5e","Type":"ContainerDied","Data":"a0266001fd8a8179a4805f9a975b2d1a81c50d0f2e9bc823f8152f37e4be4965"}
Sep 17 14:32:00.862144 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.862113    2560 generic.go:356] "Generic (PLEG): container finished" podID="3fbc4647-73e7-409c-afab-e14d82c4aeb8" containerID="90e79203442a2b4c9f806023405e77999ed4cdab45c8da7c0b7ebaa60178c144" exitCode=0
Sep 17 14:32:00.862292 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.862204    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q" event={"ID":"3fbc4647-73e7-409c-afab-e14d82c4aeb8","Type":"ContainerDied","Data":"90e79203442a2b4c9f806023405e77999ed4cdab45c8da7c0b7ebaa60178c144"}
Sep 17 14:32:00.862292 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.862229    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:32:00.862292 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.862270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q" event={"ID":"3fbc4647-73e7-409c-afab-e14d82c4aeb8","Type":"ContainerDied","Data":"af677fc132b1af4aac9d9b1db1808da4d57a665e05f4b8c2db2666aa74632cc3"}
Sep 17 14:32:00.862649 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.862294    2560 scope.go:117] "RemoveContainer" containerID="90e79203442a2b4c9f806023405e77999ed4cdab45c8da7c0b7ebaa60178c144"
Sep 17 14:32:00.864314 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.864292    2560 generic.go:356] "Generic (PLEG): container finished" podID="4843a90f-8f8f-4a04-93a0-1bb9dd20c850" containerID="ee08be6e04f1b982e6f16c3bdcd82cf1f74aa7bf7c9e75241282db6b5aa385fd" exitCode=0
Sep 17 14:32:00.864417 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.864375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq" event={"ID":"4843a90f-8f8f-4a04-93a0-1bb9dd20c850","Type":"ContainerDied","Data":"ee08be6e04f1b982e6f16c3bdcd82cf1f74aa7bf7c9e75241282db6b5aa385fd"}
Sep 17 14:32:00.894655 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.894611    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:32:00.900322 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:00.900292    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:32:01.020133 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020095    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-model-cache\") pod \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") "
Sep 17 14:32:01.020133 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020140    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tls-certs\") pod \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") "
Sep 17 14:32:01.020403 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020157    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-dshm\") pod \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") "
Sep 17 14:32:01.020403 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020180    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-home\") pod \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") "
Sep 17 14:32:01.020403 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020328    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tmp-dir\") pod \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") "
Sep 17 14:32:01.020403 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tmp-dir\") pod \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") "
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020420    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fbc4647-73e7-409c-afab-e14d82c4aeb8" (UID: "3fbc4647-73e7-409c-afab-e14d82c4aeb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020439    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-home" (OuterVolumeSpecName: "home") pod "4843a90f-8f8f-4a04-93a0-1bb9dd20c850" (UID: "4843a90f-8f8f-4a04-93a0-1bb9dd20c850"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020440    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-dshm\") pod \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") "
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020493    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-home\") pod \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") "
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020510    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-home\") pod \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") "
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020537    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-dshm\") pod \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") "
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020572    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tmp-dir\") pod \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") "
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020628    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-model-cache\") pod \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") "
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020684    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr4d9\" (UniqueName: \"kubernetes.io/projected/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-kube-api-access-kr4d9\") pod \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") "
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020720    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-home" (OuterVolumeSpecName: "home") pod "1a8a49ee-08f1-49d0-9430-0d133f216b5e" (UID: "1a8a49ee-08f1-49d0-9430-0d133f216b5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020733    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkg9c\" (UniqueName: \"kubernetes.io/projected/3fbc4647-73e7-409c-afab-e14d82c4aeb8-kube-api-access-dkg9c\") pod \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\" (UID: \"3fbc4647-73e7-409c-afab-e14d82c4aeb8\") "
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-model-cache\") pod \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") "
Sep 17 14:32:01.020819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020801    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbd75\" (UniqueName: \"kubernetes.io/projected/1a8a49ee-08f1-49d0-9430-0d133f216b5e-kube-api-access-cbd75\") pod \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") "
Sep 17 14:32:01.021505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020830    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tls-certs\") pod \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\" (UID: \"1a8a49ee-08f1-49d0-9430-0d133f216b5e\") "
Sep 17 14:32:01.021505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020874    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fbc4647-73e7-409c-afab-e14d82c4aeb8" (UID: "3fbc4647-73e7-409c-afab-e14d82c4aeb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.021505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020894    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tls-certs\") pod \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\" (UID: \"4843a90f-8f8f-4a04-93a0-1bb9dd20c850\") "
Sep 17 14:32:01.021505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.020894    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-home" (OuterVolumeSpecName: "home") pod "3fbc4647-73e7-409c-afab-e14d82c4aeb8" (UID: "3fbc4647-73e7-409c-afab-e14d82c4aeb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.021505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.021090    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4843a90f-8f8f-4a04-93a0-1bb9dd20c850" (UID: "4843a90f-8f8f-4a04-93a0-1bb9dd20c850"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.021505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.021299    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.021505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.021319    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.021505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.021332    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.022012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.021348    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.022012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.021788    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.022012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.021803    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.022012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.021529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a8a49ee-08f1-49d0-9430-0d133f216b5e" (UID: "1a8a49ee-08f1-49d0-9430-0d133f216b5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.022430 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.022405    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a8a49ee-08f1-49d0-9430-0d133f216b5e" (UID: "1a8a49ee-08f1-49d0-9430-0d133f216b5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.022520 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.022491    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-model-cache" (OuterVolumeSpecName: "model-cache") pod "4843a90f-8f8f-4a04-93a0-1bb9dd20c850" (UID: "4843a90f-8f8f-4a04-93a0-1bb9dd20c850"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.024167 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.024136    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-dshm" (OuterVolumeSpecName: "dshm") pod "4843a90f-8f8f-4a04-93a0-1bb9dd20c850" (UID: "4843a90f-8f8f-4a04-93a0-1bb9dd20c850"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.025820 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.025772    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4843a90f-8f8f-4a04-93a0-1bb9dd20c850" (UID: "4843a90f-8f8f-4a04-93a0-1bb9dd20c850"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:32:01.025938 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.025848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-dshm" (OuterVolumeSpecName: "dshm") pod "3fbc4647-73e7-409c-afab-e14d82c4aeb8" (UID: "3fbc4647-73e7-409c-afab-e14d82c4aeb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.026014 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.025871    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8a49ee-08f1-49d0-9430-0d133f216b5e-kube-api-access-cbd75" (OuterVolumeSpecName: "kube-api-access-cbd75") pod "1a8a49ee-08f1-49d0-9430-0d133f216b5e" (UID: "1a8a49ee-08f1-49d0-9430-0d133f216b5e"). InnerVolumeSpecName "kube-api-access-cbd75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:32:01.026014 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.025989    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbc4647-73e7-409c-afab-e14d82c4aeb8-kube-api-access-dkg9c" (OuterVolumeSpecName: "kube-api-access-dkg9c") pod "3fbc4647-73e7-409c-afab-e14d82c4aeb8" (UID: "3fbc4647-73e7-409c-afab-e14d82c4aeb8"). InnerVolumeSpecName "kube-api-access-dkg9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:32:01.026014 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.025848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fbc4647-73e7-409c-afab-e14d82c4aeb8" (UID: "3fbc4647-73e7-409c-afab-e14d82c4aeb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:32:01.026014 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.025965    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-dshm" (OuterVolumeSpecName: "dshm") pod "1a8a49ee-08f1-49d0-9430-0d133f216b5e" (UID: "1a8a49ee-08f1-49d0-9430-0d133f216b5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.026174 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.025964    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-kube-api-access-kr4d9" (OuterVolumeSpecName: "kube-api-access-kr4d9") pod "4843a90f-8f8f-4a04-93a0-1bb9dd20c850" (UID: "4843a90f-8f8f-4a04-93a0-1bb9dd20c850"). InnerVolumeSpecName "kube-api-access-kr4d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:32:01.026174 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.026111    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a8a49ee-08f1-49d0-9430-0d133f216b5e" (UID: "1a8a49ee-08f1-49d0-9430-0d133f216b5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:32:01.053899 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.053867    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:32:01.122569 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122533    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122569 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122562    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122569 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122571    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122581    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr4d9\" (UniqueName: \"kubernetes.io/projected/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-kube-api-access-kr4d9\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122593    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkg9c\" (UniqueName: \"kubernetes.io/projected/3fbc4647-73e7-409c-afab-e14d82c4aeb8-kube-api-access-dkg9c\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122602    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122611    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbd75\" (UniqueName: \"kubernetes.io/projected/1a8a49ee-08f1-49d0-9430-0d133f216b5e-kube-api-access-cbd75\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122619    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122628    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4843a90f-8f8f-4a04-93a0-1bb9dd20c850-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122635    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbc4647-73e7-409c-afab-e14d82c4aeb8-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122642    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbc4647-73e7-409c-afab-e14d82c4aeb8-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.122898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.122649    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8a49ee-08f1-49d0-9430-0d133f216b5e-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.223536 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.223501    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-home\") pod \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") "
Sep 17 14:32:01.223765 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.223600    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-model-cache\") pod \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") "
Sep 17 14:32:01.223765 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.223618    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tmp-dir\") pod \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") "
Sep 17 14:32:01.223765 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.223647    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-dshm\") pod \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") "
Sep 17 14:32:01.223765 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.223665    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tls-certs\") pod \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") "
Sep 17 14:32:01.223765 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.223685    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdbb4\" (UniqueName: \"kubernetes.io/projected/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-kube-api-access-xdbb4\") pod \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\" (UID: \"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c\") "
Sep 17 14:32:01.224028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.223835    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-home" (OuterVolumeSpecName: "home") pod "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" (UID: "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.224028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.223886    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" (UID: "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.224028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.223914    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" (UID: "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.224028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.224000    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.224028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.224012    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.224028 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.224021    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.225838 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.225804    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-dshm" (OuterVolumeSpecName: "dshm") pod "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" (UID: "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:01.225961 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.225840    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" (UID: "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:32:01.225961 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.225869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-kube-api-access-xdbb4" (OuterVolumeSpecName: "kube-api-access-xdbb4") pod "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" (UID: "b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c"). InnerVolumeSpecName "kube-api-access-xdbb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:32:01.324710 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.324614    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.324710 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.324651    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.324710 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.324662    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdbb4\" (UniqueName: \"kubernetes.io/projected/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c-kube-api-access-xdbb4\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:01.873862 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.873525    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"
Sep 17 14:32:01.873862 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.873548    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96" event={"ID":"1a8a49ee-08f1-49d0-9430-0d133f216b5e","Type":"ContainerDied","Data":"5e05c7ff15913c5c8a0b557623d17bac167f8139bd01245a2b031a7a7a1059df"}
Sep 17 14:32:01.873862 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.873609    2560 scope.go:117] "RemoveContainer" containerID="a0266001fd8a8179a4805f9a975b2d1a81c50d0f2e9bc823f8152f37e4be4965"
Sep 17 14:32:01.875731 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.875698    2560 generic.go:356] "Generic (PLEG): container finished" podID="b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" containerID="8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a" exitCode=0
Sep 17 14:32:01.875823 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.875794    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f" event={"ID":"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c","Type":"ContainerDied","Data":"8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a"}
Sep 17 14:32:01.875862 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.875832    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f" event={"ID":"b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c","Type":"ContainerDied","Data":"495ebee358028ef1b0a393590434a55f39d1eb56b0f71715ddf3cf57fc0b21f8"}
Sep 17 14:32:01.875915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.875895    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"
Sep 17 14:32:01.878107 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.878083    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"
Sep 17 14:32:01.882086 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.882046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq" event={"ID":"4843a90f-8f8f-4a04-93a0-1bb9dd20c850","Type":"ContainerDied","Data":"c52dff85cf8381c154614f1e109ca366ee46bc82c15123c0a60611650c41c858"}
Sep 17 14:32:01.882246 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.882133    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"
Sep 17 14:32:01.886596 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.886513    2560 scope.go:117] "RemoveContainer" containerID="8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a"
Sep 17 14:32:01.897282 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.897260    2560 scope.go:117] "RemoveContainer" containerID="8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a"
Sep 17 14:32:01.897633 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:32:01.897611    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a\": container with ID starting with 8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a not found: ID does not exist" containerID="8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a"
Sep 17 14:32:01.897727 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.897646    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a"} err="failed to get container status \"8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a\": rpc error: code = NotFound desc = could not find container \"8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a\": container with ID starting with 8a380dd73afab2b85a996fed3c95752d90f4bfa850c484c17d9f827233776d3a not found: ID does not exist"
Sep 17 14:32:01.897727 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.897669    2560 scope.go:117] "RemoveContainer" containerID="ee08be6e04f1b982e6f16c3bdcd82cf1f74aa7bf7c9e75241282db6b5aa385fd"
Sep 17 14:32:01.903387 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.903335    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"]
Sep 17 14:32:01.909052 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.909025    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-w0-1269b2-kserve-6c9c54ff97-ljk96"]
Sep 17 14:32:01.920170 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.920139    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"]
Sep 17 14:32:01.924149 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.924121    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/simulator-w0-1269b2-kserve-75b4cc9b67-rgwzq"]
Sep 17 14:32:01.934290 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.934255    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"]
Sep 17 14:32:01.940599 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.940566    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/distinct-2-w0-1269b2-kserve-6b84c79498-tkj4q"]
Sep 17 14:32:01.952915 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.952881    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"]
Sep 17 14:32:01.957839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:01.957805    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1269b2/premium-w0-1269b2-kserve-6dfc5bfbd6-kc84f"]
Sep 17 14:32:02.965150 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:02.965119    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8a49ee-08f1-49d0-9430-0d133f216b5e" path="/var/lib/kubelet/pods/1a8a49ee-08f1-49d0-9430-0d133f216b5e/volumes"
Sep 17 14:32:02.965559 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:02.965544    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fbc4647-73e7-409c-afab-e14d82c4aeb8" path="/var/lib/kubelet/pods/3fbc4647-73e7-409c-afab-e14d82c4aeb8/volumes"
Sep 17 14:32:02.965928 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:02.965915    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4843a90f-8f8f-4a04-93a0-1bb9dd20c850" path="/var/lib/kubelet/pods/4843a90f-8f8f-4a04-93a0-1bb9dd20c850/volumes"
Sep 17 14:32:02.966647 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:02.966631    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" path="/var/lib/kubelet/pods/b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c/volumes"
Sep 17 14:32:12.915319 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:12.915285    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"]
Sep 17 14:32:12.915880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:12.915627    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg" podUID="c887c738-bb81-4abe-96df-7b2ce5bac1c8" containerName="payload-pre-processing" containerID="cri-o://06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55" gracePeriod=30
Sep 17 14:32:13.168308 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.168228    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:32:13.240935 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.240897    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c887c738-bb81-4abe-96df-7b2ce5bac1c8-plugins-config-volume\") pod \"c887c738-bb81-4abe-96df-7b2ce5bac1c8\" (UID: \"c887c738-bb81-4abe-96df-7b2ce5bac1c8\") "
Sep 17 14:32:13.241175 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.241023    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdbw4\" (UniqueName: \"kubernetes.io/projected/c887c738-bb81-4abe-96df-7b2ce5bac1c8-kube-api-access-hdbw4\") pod \"c887c738-bb81-4abe-96df-7b2ce5bac1c8\" (UID: \"c887c738-bb81-4abe-96df-7b2ce5bac1c8\") "
Sep 17 14:32:13.241319 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.241286    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c887c738-bb81-4abe-96df-7b2ce5bac1c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c887c738-bb81-4abe-96df-7b2ce5bac1c8" (UID: "c887c738-bb81-4abe-96df-7b2ce5bac1c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:32:13.243186 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.243155    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c887c738-bb81-4abe-96df-7b2ce5bac1c8-kube-api-access-hdbw4" (OuterVolumeSpecName: "kube-api-access-hdbw4") pod "c887c738-bb81-4abe-96df-7b2ce5bac1c8" (UID: "c887c738-bb81-4abe-96df-7b2ce5bac1c8"). InnerVolumeSpecName "kube-api-access-hdbw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:32:13.341848 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.341819    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c887c738-bb81-4abe-96df-7b2ce5bac1c8-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:13.341848 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.341847    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdbw4\" (UniqueName: \"kubernetes.io/projected/c887c738-bb81-4abe-96df-7b2ce5bac1c8-kube-api-access-hdbw4\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:32:13.949237 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.949203    2560 generic.go:356] "Generic (PLEG): container finished" podID="c887c738-bb81-4abe-96df-7b2ce5bac1c8" containerID="06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55" exitCode=0
Sep 17 14:32:13.949646 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.949243    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg" event={"ID":"c887c738-bb81-4abe-96df-7b2ce5bac1c8","Type":"ContainerDied","Data":"06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55"}
Sep 17 14:32:13.949646 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.949289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg" event={"ID":"c887c738-bb81-4abe-96df-7b2ce5bac1c8","Type":"ContainerDied","Data":"6901461cdf82415ce3720ce312f4af9aa20cbefdec914a30b423f897058baeb0"}
Sep 17 14:32:13.949646 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.949311    2560 scope.go:117] "RemoveContainer" containerID="06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55"
Sep 17 14:32:13.949646 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.949245    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"
Sep 17 14:32:13.958414 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.958395    2560 scope.go:117] "RemoveContainer" containerID="06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55"
Sep 17 14:32:13.958743 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:32:13.958715    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55\": container with ID starting with 06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55 not found: ID does not exist" containerID="06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55"
Sep 17 14:32:13.958743 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.958757    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55"} err="failed to get container status \"06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55\": rpc error: code = NotFound desc = could not find container \"06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55\": container with ID starting with 06158a4c034fcc947667318220208400debac3a342797d516daafb0824b36d55 not found: ID does not exist"
Sep 17 14:32:13.971012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.970975    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"]
Sep 17 14:32:13.975532 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:13.975500    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b83f70-76b45865f4-cqtjg"]
Sep 17 14:32:14.965735 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:32:14.965706    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c887c738-bb81-4abe-96df-7b2ce5bac1c8" path="/var/lib/kubelet/pods/c887c738-bb81-4abe-96df-7b2ce5bac1c8/volumes"
Sep 17 14:33:02.408663 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:02.408627    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"]
Sep 17 14:33:02.436725 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:02.436694    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"]
Sep 17 14:33:13.383995 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:13.383937    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"]
Sep 17 14:33:13.384505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:13.384351    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf" podUID="6c78eaec-b277-45e8-8518-63889f408020" containerName="payload-pre-processing" containerID="cri-o://c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5" gracePeriod=30
Sep 17 14:33:13.645931 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:13.645856    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:33:13.727113 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:13.727072    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grl2j\" (UniqueName: \"kubernetes.io/projected/6c78eaec-b277-45e8-8518-63889f408020-kube-api-access-grl2j\") pod \"6c78eaec-b277-45e8-8518-63889f408020\" (UID: \"6c78eaec-b277-45e8-8518-63889f408020\") "
Sep 17 14:33:13.727290 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:13.727163    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c78eaec-b277-45e8-8518-63889f408020-plugins-config-volume\") pod \"6c78eaec-b277-45e8-8518-63889f408020\" (UID: \"6c78eaec-b277-45e8-8518-63889f408020\") "
Sep 17 14:33:13.727559 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:13.727537    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c78eaec-b277-45e8-8518-63889f408020-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c78eaec-b277-45e8-8518-63889f408020" (UID: "6c78eaec-b277-45e8-8518-63889f408020"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:33:13.729220 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:13.729193    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c78eaec-b277-45e8-8518-63889f408020-kube-api-access-grl2j" (OuterVolumeSpecName: "kube-api-access-grl2j") pod "6c78eaec-b277-45e8-8518-63889f408020" (UID: "6c78eaec-b277-45e8-8518-63889f408020"). InnerVolumeSpecName "kube-api-access-grl2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:33:13.828860 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:13.828818    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grl2j\" (UniqueName: \"kubernetes.io/projected/6c78eaec-b277-45e8-8518-63889f408020-kube-api-access-grl2j\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:13.828860 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:13.828852    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c78eaec-b277-45e8-8518-63889f408020-plugins-config-volume\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:14.247339 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:14.247301    2560 generic.go:356] "Generic (PLEG): container finished" podID="6c78eaec-b277-45e8-8518-63889f408020" containerID="c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5" exitCode=0
Sep 17 14:33:14.247544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:14.247347    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"
Sep 17 14:33:14.247544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:14.247371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf" event={"ID":"6c78eaec-b277-45e8-8518-63889f408020","Type":"ContainerDied","Data":"c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5"}
Sep 17 14:33:14.247544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:14.247429    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf" event={"ID":"6c78eaec-b277-45e8-8518-63889f408020","Type":"ContainerDied","Data":"062d8db42d650019d58da75b222356cf5b45f5516bf731872998002d540ba726"}
Sep 17 14:33:14.247544 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:14.247454    2560 scope.go:117] "RemoveContainer" containerID="c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5"
Sep 17 14:33:14.256335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:14.256303    2560 scope.go:117] "RemoveContainer" containerID="c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5"
Sep 17 14:33:14.256668 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:33:14.256645    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5\": container with ID starting with c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5 not found: ID does not exist" containerID="c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5"
Sep 17 14:33:14.256732 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:14.256677    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5"} err="failed to get container status \"c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5\": rpc error: code = NotFound desc = could not find container \"c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5\": container with ID starting with c6d2fc3b738a21d2bf0acfa9914dd6fbcf21169759a59f03559fb5fc441d98e5 not found: ID does not exist"
Sep 17 14:33:14.268342 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:14.268311    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"]
Sep 17 14:33:14.274330 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:14.274303    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fcd9f3-75484f6654-8cpcf"]
Sep 17 14:33:14.965882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:14.965849    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c78eaec-b277-45e8-8518-63889f408020" path="/var/lib/kubelet/pods/6c78eaec-b277-45e8-8518-63889f408020/volumes"
Sep 17 14:33:17.428081 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.428040    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk" podUID="865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" containerName="main" containerID="cri-o://087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9" gracePeriod=45
Sep 17 14:33:17.455770 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.455731    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2" podUID="d08f7a56-5408-4885-a717-c985fc5f911f" containerName="main" containerID="cri-o://41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a" gracePeriod=45
Sep 17 14:33:17.699240 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.699211    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:33:17.729336 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.729309    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:33:17.869711 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.869672    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5j9z\" (UniqueName: \"kubernetes.io/projected/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-kube-api-access-g5j9z\") pod \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") "
Sep 17 14:33:17.869916 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.869736    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08f7a56-5408-4885-a717-c985fc5f911f-tls-certs\") pod \"d08f7a56-5408-4885-a717-c985fc5f911f\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") "
Sep 17 14:33:17.869916 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.869759    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tmp-dir\") pod \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") "
Sep 17 14:33:17.869916 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.869794    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p29qw\" (UniqueName: \"kubernetes.io/projected/d08f7a56-5408-4885-a717-c985fc5f911f-kube-api-access-p29qw\") pod \"d08f7a56-5408-4885-a717-c985fc5f911f\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") "
Sep 17 14:33:17.869916 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.869820    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-model-cache\") pod \"d08f7a56-5408-4885-a717-c985fc5f911f\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") "
Sep 17 14:33:17.869916 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.869851    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tls-certs\") pod \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") "
Sep 17 14:33:17.869916 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.869900    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-dshm\") pod \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") "
Sep 17 14:33:17.870252 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.869945    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-model-cache\") pod \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") "
Sep 17 14:33:17.870252 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.869972    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-tmp-dir\") pod \"d08f7a56-5408-4885-a717-c985fc5f911f\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") "
Sep 17 14:33:17.870252 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.869992    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-home\") pod \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\" (UID: \"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8\") "
Sep 17 14:33:17.870252 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870045    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-dshm\") pod \"d08f7a56-5408-4885-a717-c985fc5f911f\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") "
Sep 17 14:33:17.870252 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870056    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" (UID: "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:17.870252 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870078    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-home\") pod \"d08f7a56-5408-4885-a717-c985fc5f911f\" (UID: \"d08f7a56-5408-4885-a717-c985fc5f911f\") "
Sep 17 14:33:17.870252 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870109    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d08f7a56-5408-4885-a717-c985fc5f911f" (UID: "d08f7a56-5408-4885-a717-c985fc5f911f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:17.870633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870286    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d08f7a56-5408-4885-a717-c985fc5f911f" (UID: "d08f7a56-5408-4885-a717-c985fc5f911f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:17.870633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870476    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-home" (OuterVolumeSpecName: "home") pod "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" (UID: "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:17.870633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870485    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.870633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-home" (OuterVolumeSpecName: "home") pod "d08f7a56-5408-4885-a717-c985fc5f911f" (UID: "d08f7a56-5408-4885-a717-c985fc5f911f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:17.870633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870524    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.870633 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870541    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.870928 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.870715    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" (UID: "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:17.872734 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.872690    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08f7a56-5408-4885-a717-c985fc5f911f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d08f7a56-5408-4885-a717-c985fc5f911f" (UID: "d08f7a56-5408-4885-a717-c985fc5f911f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:33:17.872734 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.872691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" (UID: "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:33:17.872898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.872753    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-dshm" (OuterVolumeSpecName: "dshm") pod "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" (UID: "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:17.872898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.872766    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-dshm" (OuterVolumeSpecName: "dshm") pod "d08f7a56-5408-4885-a717-c985fc5f911f" (UID: "d08f7a56-5408-4885-a717-c985fc5f911f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:17.872898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.872794    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-kube-api-access-g5j9z" (OuterVolumeSpecName: "kube-api-access-g5j9z") pod "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" (UID: "865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8"). InnerVolumeSpecName "kube-api-access-g5j9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:33:17.872898 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.872818    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08f7a56-5408-4885-a717-c985fc5f911f-kube-api-access-p29qw" (OuterVolumeSpecName: "kube-api-access-p29qw") pod "d08f7a56-5408-4885-a717-c985fc5f911f" (UID: "d08f7a56-5408-4885-a717-c985fc5f911f"). InnerVolumeSpecName "kube-api-access-p29qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:33:17.971819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.971730    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.971819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.971762    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.971819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.971771    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.971819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.971781    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.971819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.971788    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08f7a56-5408-4885-a717-c985fc5f911f-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.971819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.971797    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5j9z\" (UniqueName: \"kubernetes.io/projected/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-kube-api-access-g5j9z\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.971819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.971805    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08f7a56-5408-4885-a717-c985fc5f911f-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.971819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.971815    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p29qw\" (UniqueName: \"kubernetes.io/projected/d08f7a56-5408-4885-a717-c985fc5f911f-kube-api-access-p29qw\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:17.971819 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:17.971823    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:33:18.267726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.267686    2560 generic.go:356] "Generic (PLEG): container finished" podID="865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" containerID="087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9" exitCode=0
Sep 17 14:33:18.267726 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.267731    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"
Sep 17 14:33:18.267977 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.267771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk" event={"ID":"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8","Type":"ContainerDied","Data":"087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9"}
Sep 17 14:33:18.267977 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.267815    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk" event={"ID":"865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8","Type":"ContainerDied","Data":"53dd403bf576de45b66034ee7636989c36a55afa5b93e5b59c10b612fab1fdbd"}
Sep 17 14:33:18.267977 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.267833    2560 scope.go:117] "RemoveContainer" containerID="087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9"
Sep 17 14:33:18.269088 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.269071    2560 generic.go:356] "Generic (PLEG): container finished" podID="d08f7a56-5408-4885-a717-c985fc5f911f" containerID="41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a" exitCode=0
Sep 17 14:33:18.269180 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.269099    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2" event={"ID":"d08f7a56-5408-4885-a717-c985fc5f911f","Type":"ContainerDied","Data":"41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a"}
Sep 17 14:33:18.269180 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.269118    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"
Sep 17 14:33:18.269180 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.269131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2" event={"ID":"d08f7a56-5408-4885-a717-c985fc5f911f","Type":"ContainerDied","Data":"8ad005ad7a77c7d930ad040eb47c5ad2093c005f50412e246716e5c2a9352302"}
Sep 17 14:33:18.279437 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.279407    2560 scope.go:117] "RemoveContainer" containerID="087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9"
Sep 17 14:33:18.280053 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:33:18.280026    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9\": container with ID starting with 087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9 not found: ID does not exist" containerID="087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9"
Sep 17 14:33:18.280162 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.280060    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9"} err="failed to get container status \"087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9\": rpc error: code = NotFound desc = could not find container \"087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9\": container with ID starting with 087ddc2d8af364446dd835998695bed479650d52f6f5e4efab6973211b6aa6b9 not found: ID does not exist"
Sep 17 14:33:18.280162 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.280077    2560 scope.go:117] "RemoveContainer" containerID="41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a"
Sep 17 14:33:18.287847 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.287824    2560 scope.go:117] "RemoveContainer" containerID="41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a"
Sep 17 14:33:18.288107 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:33:18.288090    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a\": container with ID starting with 41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a not found: ID does not exist" containerID="41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a"
Sep 17 14:33:18.288174 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.288113    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a"} err="failed to get container status \"41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a\": rpc error: code = NotFound desc = could not find container \"41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a\": container with ID starting with 41f661678b24807d326a5bc95bfb039db0b21a851f63499c9f5532c18164478a not found: ID does not exist"
Sep 17 14:33:18.296136 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.296112    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"]
Sep 17 14:33:18.304648 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.304620    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/distinct-2-w4-fcd9f3-kserve-c99d795bc-h2brk"]
Sep 17 14:33:18.313728 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.313702    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"]
Sep 17 14:33:18.318128 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.318101    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fcd9f3/simulator-w4-fcd9f3-kserve-fff6dc848-fg9n2"]
Sep 17 14:33:18.965788 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.965754    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" path="/var/lib/kubelet/pods/865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8/volumes"
Sep 17 14:33:18.966166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:33:18.966152    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08f7a56-5408-4885-a717-c985fc5f911f" path="/var/lib/kubelet/pods/d08f7a56-5408-4885-a717-c985fc5f911f/volumes"
Sep 17 14:33:50.498669 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:33:50.498623    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:33:40Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:33:40Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:33:40Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:33:40Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-131-146.ec2.internal\": Patch \"https://a2b89f3df576c40229a4e99950687ee2-0e02d3b4ed87b95e.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-131-146.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 17 14:36:00.889062 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889029    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"]
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889384    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad04203b-e2ad-4d57-bb53-46c43dfbc242" containerName="payload-pre-processing"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889398    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad04203b-e2ad-4d57-bb53-46c43dfbc242" containerName="payload-pre-processing"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889410    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c78eaec-b277-45e8-8518-63889f408020" containerName="payload-pre-processing"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889416    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c78eaec-b277-45e8-8518-63889f408020" containerName="payload-pre-processing"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889422    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08f7a56-5408-4885-a717-c985fc5f911f" containerName="main"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889429    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08f7a56-5408-4885-a717-c985fc5f911f" containerName="main"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889436    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4843a90f-8f8f-4a04-93a0-1bb9dd20c850" containerName="main"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889442    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4843a90f-8f8f-4a04-93a0-1bb9dd20c850" containerName="main"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889451    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e56b1f-513e-4c07-a8d7-5d84406ae6d7" containerName="payload-pre-processing"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889457    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e56b1f-513e-4c07-a8d7-5d84406ae6d7" containerName="payload-pre-processing"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889464    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a339aeec-c248-4a1d-bcfe-e4ba36b2c099" containerName="payload-processing"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889469    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a339aeec-c248-4a1d-bcfe-e4ba36b2c099" containerName="payload-processing"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889474    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87254986-487e-4ac8-8c31-9ba553408fea" containerName="maas-api"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889479    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87254986-487e-4ac8-8c31-9ba553408fea" containerName="maas-api"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889485    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8a49ee-08f1-49d0-9430-0d133f216b5e" containerName="main"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889489    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8a49ee-08f1-49d0-9430-0d133f216b5e" containerName="main"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889497    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d81f3d-b91a-4cac-8670-06e6d2c64876" containerName="maas-api"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889502    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d81f3d-b91a-4cac-8670-06e6d2c64876" containerName="maas-api"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889508    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" containerName="main"
Sep 17 14:36:00.889502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889513    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889518    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c887c738-bb81-4abe-96df-7b2ce5bac1c8" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889523    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c887c738-bb81-4abe-96df-7b2ce5bac1c8" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889529    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b696cf-fed3-42da-9bc3-47be4a72ac56" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889534    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b696cf-fed3-42da-9bc3-47be4a72ac56" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889540    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b6121a-8b7d-4c16-8573-77e1205a1ba4" containerName="maas-api"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889544    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b6121a-8b7d-4c16-8573-77e1205a1ba4" containerName="maas-api"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889549    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbc4647-73e7-409c-afab-e14d82c4aeb8" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889554    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbc4647-73e7-409c-afab-e14d82c4aeb8" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889561    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889566    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889572    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03cede8-2406-458f-906b-e58c41d0e39f" containerName="maas-api"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889577    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03cede8-2406-458f-906b-e58c41d0e39f" containerName="maas-api"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889585    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d908727-b228-420f-9118-ff8ac0d56b71" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889590    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d908727-b228-420f-9118-ff8ac0d56b71" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889642    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="865fe8b2-4ed4-44f5-9b87-d3ec41aeb1d8" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889650    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d5aab3-e6bc-4c69-9a1d-25c6bf40a59c" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889657    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbc4647-73e7-409c-afab-e14d82c4aeb8" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889663    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d908727-b228-420f-9118-ff8ac0d56b71" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889670    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e56b1f-513e-4c07-a8d7-5d84406ae6d7" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889675    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87254986-487e-4ac8-8c31-9ba553408fea" containerName="maas-api"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889681    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c78eaec-b277-45e8-8518-63889f408020" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889686    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad04203b-e2ad-4d57-bb53-46c43dfbc242" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889690    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c887c738-bb81-4abe-96df-7b2ce5bac1c8" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889696    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03cede8-2406-458f-906b-e58c41d0e39f" containerName="maas-api"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889700    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8a49ee-08f1-49d0-9430-0d133f216b5e" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889705    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="14b6121a-8b7d-4c16-8573-77e1205a1ba4" containerName="maas-api"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889709    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4843a90f-8f8f-4a04-93a0-1bb9dd20c850" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889713    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08f7a56-5408-4885-a717-c985fc5f911f" containerName="main"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889717    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="72d81f3d-b91a-4cac-8670-06e6d2c64876" containerName="maas-api"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889722    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0b696cf-fed3-42da-9bc3-47be4a72ac56" containerName="payload-pre-processing"
Sep 17 14:36:00.890099 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.889727    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a339aeec-c248-4a1d-bcfe-e4ba36b2c099" containerName="payload-processing"
Sep 17 14:36:00.892565 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.892548    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:00.896279 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.896255    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc3801\"/\"default-dockercfg-h86g7\""
Sep 17 14:36:00.896279 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.896274    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc3801\"/\"openshift-service-ca.crt\""
Sep 17 14:36:00.896701 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.896679    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-self-signed-certs\""
Sep 17 14:36:00.896805 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:00.896758    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc3801\"/\"kube-root-ca.crt\""
Sep 17 14:36:00.905235 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:36:00.905188    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.014420 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.014344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tmp-dir\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.014632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.014479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-model-cache\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.014632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.014528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-home\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.014632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.014586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tls-certs\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.014809 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.014639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-dshm\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.014809 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.014717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlssj\" (UniqueName: \"kubernetes.io/projected/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-kube-api-access-zlssj\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.115391 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.115344    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlssj\" (UniqueName: \"kubernetes.io/projected/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-kube-api-access-zlssj\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.115586 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.115417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tmp-dir\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.115586 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.115464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-model-cache\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.115586 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.115484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-home\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.115586 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.115516    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tls-certs\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.115586 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.115550    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-dshm\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.115948 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.115925    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tmp-dir\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.116010 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.115949    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-model-cache\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.116077 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.116060    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-home\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.117738 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.117715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-dshm\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.118030 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.118009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tls-certs\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.124016 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.123988    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlssj\" (UniqueName: \"kubernetes.io/projected/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-kube-api-access-zlssj\") pod \"sub-test-model-cc3801-kserve-58769c6fd7-647d8\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:01.202293 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:01.202183    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:02.976055 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:36:02.975992    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:03.214072 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:36:03.214038    2560 scope.go:117] "RemoveContainer" containerID="59293c186aa7f65cd0329dcbfcb6b63851d5dc9f5089305fcbdb01f6900d5cc2"
Sep 17 14:36:12.973329 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:36:12.973251    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:22.974794 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:36:22.974719    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:32.974901 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:36:32.974786    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:42.976268 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:36:42.976214    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:36:52.974210 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:36:52.974154    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:37:02.972583 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:37:02.972527    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:37:12.973852 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:37:12.973797    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:37:22.974252 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:37:22.974198    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:37:32.974537 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:37:32.974480    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:37:42.973407 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:37:42.973328    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:37:52.975991 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:37:52.975933    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:38:01.406463 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:01.406409    2560 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 17 14:38:01.406463 ip-10-0-131-146 kubenswrapper[2560]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_sub-test-model-cc3801-kserve-58769c6fd7-647d8_ai-tenant-e2e-shared-a-w3-cc3801_3fe1664f-f4d0-44c3-9a18-f1bef45050c0_0(5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39): error adding pod ai-tenant-e2e-shared-a-w3-cc3801_sub-test-model-cc3801-kserve-58769c6fd7-647d8 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39" Netns:"/var/run/netns/8e3d0370-fda0-4050-801b-7283a5d297ae" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-shared-a-w3-cc3801;K8S_POD_NAME=sub-test-model-cc3801-kserve-58769c6fd7-647d8;K8S_POD_INFRA_CONTAINER_ID=5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39;K8S_POD_UID=3fe1664f-f4d0-44c3-9a18-f1bef45050c0" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8] networking: [ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8/3fe1664f-f4d0-44c3-9a18-f1bef45050c0:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8 5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39 network default NAD default NAD key default] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded
Sep 17 14:38:01.406463 ip-10-0-131-146 kubenswrapper[2560]:         '
Sep 17 14:38:01.406463 ip-10-0-131-146 kubenswrapper[2560]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 17 14:38:01.406463 ip-10-0-131-146 kubenswrapper[2560]:  >
Sep 17 14:38:01.407011 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:01.406496    2560 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 17 14:38:01.407011 ip-10-0-131-146 kubenswrapper[2560]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_sub-test-model-cc3801-kserve-58769c6fd7-647d8_ai-tenant-e2e-shared-a-w3-cc3801_3fe1664f-f4d0-44c3-9a18-f1bef45050c0_0(5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39): error adding pod ai-tenant-e2e-shared-a-w3-cc3801_sub-test-model-cc3801-kserve-58769c6fd7-647d8 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39" Netns:"/var/run/netns/8e3d0370-fda0-4050-801b-7283a5d297ae" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-shared-a-w3-cc3801;K8S_POD_NAME=sub-test-model-cc3801-kserve-58769c6fd7-647d8;K8S_POD_INFRA_CONTAINER_ID=5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39;K8S_POD_UID=3fe1664f-f4d0-44c3-9a18-f1bef45050c0" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8] networking: [ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8/3fe1664f-f4d0-44c3-9a18-f1bef45050c0:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8 5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39 network default NAD default NAD key default] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded
Sep 17 14:38:01.407011 ip-10-0-131-146 kubenswrapper[2560]:         '
Sep 17 14:38:01.407011 ip-10-0-131-146 kubenswrapper[2560]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 17 14:38:01.407011 ip-10-0-131-146 kubenswrapper[2560]:  > pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:38:01.407011 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:01.406522    2560 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Sep 17 14:38:01.407011 ip-10-0-131-146 kubenswrapper[2560]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_sub-test-model-cc3801-kserve-58769c6fd7-647d8_ai-tenant-e2e-shared-a-w3-cc3801_3fe1664f-f4d0-44c3-9a18-f1bef45050c0_0(5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39): error adding pod ai-tenant-e2e-shared-a-w3-cc3801_sub-test-model-cc3801-kserve-58769c6fd7-647d8 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39" Netns:"/var/run/netns/8e3d0370-fda0-4050-801b-7283a5d297ae" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-shared-a-w3-cc3801;K8S_POD_NAME=sub-test-model-cc3801-kserve-58769c6fd7-647d8;K8S_POD_INFRA_CONTAINER_ID=5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39;K8S_POD_UID=3fe1664f-f4d0-44c3-9a18-f1bef45050c0" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8] networking: [ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8/3fe1664f-f4d0-44c3-9a18-f1bef45050c0:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8 5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39 network default NAD default NAD key default] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded
Sep 17 14:38:01.407011 ip-10-0-131-146 kubenswrapper[2560]:         '
Sep 17 14:38:01.407011 ip-10-0-131-146 kubenswrapper[2560]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 17 14:38:01.407011 ip-10-0-131-146 kubenswrapper[2560]:  > pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:38:01.407313 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:01.406645    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"sub-test-model-cc3801-kserve-58769c6fd7-647d8_ai-tenant-e2e-shared-a-w3-cc3801(3fe1664f-f4d0-44c3-9a18-f1bef45050c0)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"sub-test-model-cc3801-kserve-58769c6fd7-647d8_ai-tenant-e2e-shared-a-w3-cc3801(3fe1664f-f4d0-44c3-9a18-f1bef45050c0)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_sub-test-model-cc3801-kserve-58769c6fd7-647d8_ai-tenant-e2e-shared-a-w3-cc3801_3fe1664f-f4d0-44c3-9a18-f1bef45050c0_0(5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39): error adding pod ai-tenant-e2e-shared-a-w3-cc3801_sub-test-model-cc3801-kserve-58769c6fd7-647d8 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39\\\" Netns:\\\"/var/run/netns/8e3d0370-fda0-4050-801b-7283a5d297ae\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-shared-a-w3-cc3801;K8S_POD_NAME=sub-test-model-cc3801-kserve-58769c6fd7-647d8;K8S_POD_INFRA_CONTAINER_ID=5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39;K8S_POD_UID=3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\" Path:\\\"\\\" ERRORED: error configuring pod [ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8] networking: [ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8/3fe1664f-f4d0-44c3-9a18-f1bef45050c0:ovn-kubernetes]: error adding container to network \\\"ovn-kubernetes\\\": CNI request failed with status 400: '[ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8 5227ae26774b2e556b81ce391836ebfa1126a7536ecc67aee1c14a8f05463b39 network default NAD default NAD key default] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded\\n'\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8" podUID="3fe1664f-f4d0-44c3-9a18-f1bef45050c0"
Sep 17 14:38:02.974386 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:02.974316    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:38:12.979849 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:12.979790    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:38:13.959105 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:38:13.959068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:38:13.959499 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:38:13.959481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:38:22.974542 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:22.974487    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:38:26.354310 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:26.354259    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-131-146.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/node?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused"
Sep 17 14:38:26.369621 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:26.369576    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-131-146.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/node?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused"
Sep 17 14:38:26.385054 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:26.385014    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-131-146.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/node?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused"
Sep 17 14:38:26.401271 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:26.401227    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-131-146.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/node?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused"
Sep 17 14:38:26.415545 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:26.415498    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-17T14:38:26Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-131-146.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/node?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused"
Sep 17 14:38:26.415545 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:26.415536    2560 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 17 14:38:32.974119 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:32.974064    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:38:42.972066 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:42.972006    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:38:52.975841 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:38:52.975777    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:39:02.975096 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:39:02.975038    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:39:12.973243 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:39:12.973187    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:39:22.974072 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:39:22.974014    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:39:33.177112 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:39:33.177009    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:39:42.977611 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:39:42.977526    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:39:52.974735 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:39:52.974655    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:40:02.976330 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:40:02.976271    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-17T14:36:00Z\\\",\\\"message\\\":\\\"containers with unready status: [main]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"main\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/home\\\",\\\"name\\\":\\\"home\\\"},{\\\"mountPath\\\":\\\"/tmp\\\",\\\"name\\\":\\\"tmp-dir\\\"},{\\\"mountPath\\\":\\\"/dev/shm\\\",\\\"name\\\":\\\"dshm\\\"},{\\\"mountPath\\\":\\\"/models\\\",\\\"name\\\":\\\"model-cache\\\"},{\\\"mountPath\\\":\\\"/var/run/kserve/tls\\\",\\\"name\\\":\\\"tls-certs\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zlssj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.146\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.146\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-17T14:36:00Z\\\"}}\" for pod \"ai-tenant-e2e-shared-a-w3-cc3801\"/\"sub-test-model-cc3801-kserve-58769c6fd7-647d8\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-6f4d9a9c-b466-48fc-bb29-13cef4544b0c.svc:9743/pod?timeout=10s\": dial tcp 172.30.162.63:9743: connect: connection refused" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:40:13.002456 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:40:13.002415    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"]
Sep 17 14:40:13.283396 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:40:13.283339    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"]
Sep 17 14:40:13.286616 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:40:13.286575    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fe1664f_f4d0_44c3_9a18_f1bef45050c0.slice/crio-b4d3d1c34a002907fc7732c36930b831c580544cdc66d943005d848dcfaa7943 WatchSource:0}: Error finding container b4d3d1c34a002907fc7732c36930b831c580544cdc66d943005d848dcfaa7943: Status 404 returned error can't find the container with id b4d3d1c34a002907fc7732c36930b831c580544cdc66d943005d848dcfaa7943
Sep 17 14:40:14.180632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:40:14.180598    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8" event={"ID":"3fe1664f-f4d0-44c3-9a18-f1bef45050c0","Type":"ContainerStarted","Data":"5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f"}
Sep 17 14:40:14.181092 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:40:14.180640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8" event={"ID":"3fe1664f-f4d0-44c3-9a18-f1bef45050c0","Type":"ContainerStarted","Data":"b4d3d1c34a002907fc7732c36930b831c580544cdc66d943005d848dcfaa7943"}
Sep 17 14:40:14.199999 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:40:14.199946    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8" podStartSLOduration=380.199929958 podStartE2EDuration="6m20.199929958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:40:14.199308117 +0000 UTC m=+2111.790483878" watchObservedRunningTime="2026-09-17 14:40:14.199929958 +0000 UTC m=+2111.791105719"
Sep 17 14:40:21.203143 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:40:21.203095    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:40:21.203586 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:40:21.203171    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:40:21.215405 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:40:21.215376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:40:21.226537 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:40:21.226510    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:42:43.937186 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:43.937105    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"]
Sep 17 14:42:43.940465 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:43.940446    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:43.944816 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:43.944798    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc3801\"/\"rate-test-model-cc3801-kserve-self-signed-certs\""
Sep 17 14:42:43.964587 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:43.964560    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"]
Sep 17 14:42:43.989406 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:43.989350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-tmp-dir\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:43.989578 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:43.989463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-home\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:43.989578 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:43.989509    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d09a334-e389-47b0-901f-a43ad3f8836d-tls-certs\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:43.989578 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:43.989534    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-dshm\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:43.989578 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:43.989554    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cphv\" (UniqueName: \"kubernetes.io/projected/5d09a334-e389-47b0-901f-a43ad3f8836d-kube-api-access-6cphv\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:43.989788 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:43.989663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-model-cache\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.091031 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.090990    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-model-cache\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.091188 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.091061    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-tmp-dir\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.091188 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.091098    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-home\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.091188 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.091121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d09a334-e389-47b0-901f-a43ad3f8836d-tls-certs\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.091188 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.091144    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-dshm\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.091188 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.091161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cphv\" (UniqueName: \"kubernetes.io/projected/5d09a334-e389-47b0-901f-a43ad3f8836d-kube-api-access-6cphv\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.091531 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.091507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-model-cache\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.091591 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.091516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-tmp-dir\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.091632 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.091590    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-home\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.093465 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.093444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-dshm\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.093835 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.093814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d09a334-e389-47b0-901f-a43ad3f8836d-tls-certs\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.102757 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.102735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cphv\" (UniqueName: \"kubernetes.io/projected/5d09a334-e389-47b0-901f-a43ad3f8836d-kube-api-access-6cphv\") pod \"rate-test-model-cc3801-kserve-79f988d547-nr2wt\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") " pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.248830 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.248798    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:44.531233 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.531205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"]
Sep 17 14:42:44.533199 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:42:44.533174    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d09a334_e389_47b0_901f_a43ad3f8836d.slice/crio-af386939e890cbbab7b429e820d240c0ea26d56b1b9c15b4ba676206766133aa WatchSource:0}: Error finding container af386939e890cbbab7b429e820d240c0ea26d56b1b9c15b4ba676206766133aa: Status 404 returned error can't find the container with id af386939e890cbbab7b429e820d240c0ea26d56b1b9c15b4ba676206766133aa
Sep 17 14:42:44.860260 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.860165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt" event={"ID":"5d09a334-e389-47b0-901f-a43ad3f8836d","Type":"ContainerStarted","Data":"f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b"}
Sep 17 14:42:44.860260 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.860206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt" event={"ID":"5d09a334-e389-47b0-901f-a43ad3f8836d","Type":"ContainerStarted","Data":"af386939e890cbbab7b429e820d240c0ea26d56b1b9c15b4ba676206766133aa"}
Sep 17 14:42:44.890253 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:44.890197    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt" podStartSLOduration=340.890180266 podStartE2EDuration="5m40.890180266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:42:44.888891928 +0000 UTC m=+2262.480067691" watchObservedRunningTime="2026-09-17 14:42:44.890180266 +0000 UTC m=+2262.481356027"
Sep 17 14:42:54.249248 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:54.249216    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:54.249671 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:54.249493    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:54.261956 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:54.261935    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:54.915967 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:54.915934    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:42:55.679225 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:55.674833    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"]
Sep 17 14:42:55.680512 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:42:55.680484    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"]
Sep 17 14:43:10.699424 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:10.699351    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8" podUID="3fe1664f-f4d0-44c3-9a18-f1bef45050c0" containerName="main" containerID="cri-o://5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f" gracePeriod=45
Sep 17 14:43:10.947821 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:10.947794    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:43:10.982447 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:10.982397    2560 generic.go:356] "Generic (PLEG): container finished" podID="3fe1664f-f4d0-44c3-9a18-f1bef45050c0" containerID="5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f" exitCode=0
Sep 17 14:43:10.982628 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:10.982516    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"
Sep 17 14:43:10.982628 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:10.982529    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8" event={"ID":"3fe1664f-f4d0-44c3-9a18-f1bef45050c0","Type":"ContainerDied","Data":"5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f"}
Sep 17 14:43:10.982628 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:10.982569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8" event={"ID":"3fe1664f-f4d0-44c3-9a18-f1bef45050c0","Type":"ContainerDied","Data":"b4d3d1c34a002907fc7732c36930b831c580544cdc66d943005d848dcfaa7943"}
Sep 17 14:43:10.982628 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:10.982592    2560 scope.go:117] "RemoveContainer" containerID="5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f"
Sep 17 14:43:10.991523 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:10.991503    2560 scope.go:117] "RemoveContainer" containerID="5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f"
Sep 17 14:43:10.991864 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:43:10.991840    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f\": container with ID starting with 5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f not found: ID does not exist" containerID="5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f"
Sep 17 14:43:10.991943 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:10.991873    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f"} err="failed to get container status \"5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f\": rpc error: code = NotFound desc = could not find container \"5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f\": container with ID starting with 5222bb596f891d9a652c4fef08548c3d8fb5f588b78f7584dcc27bb181f2845f not found: ID does not exist"
Sep 17 14:43:11.048499 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.048461    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tls-certs\") pod \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") "
Sep 17 14:43:11.048685 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.048523    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlssj\" (UniqueName: \"kubernetes.io/projected/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-kube-api-access-zlssj\") pod \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") "
Sep 17 14:43:11.048685 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.048547    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-model-cache\") pod \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") "
Sep 17 14:43:11.048685 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.048573    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tmp-dir\") pod \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") "
Sep 17 14:43:11.048685 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.048593    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-dshm\") pod \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") "
Sep 17 14:43:11.048685 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.048625    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-home\") pod \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\" (UID: \"3fe1664f-f4d0-44c3-9a18-f1bef45050c0\") "
Sep 17 14:43:11.048970 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.048845    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fe1664f-f4d0-44c3-9a18-f1bef45050c0" (UID: "3fe1664f-f4d0-44c3-9a18-f1bef45050c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:11.048970 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.048920    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fe1664f-f4d0-44c3-9a18-f1bef45050c0" (UID: "3fe1664f-f4d0-44c3-9a18-f1bef45050c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:11.049059 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.048996    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-home" (OuterVolumeSpecName: "home") pod "3fe1664f-f4d0-44c3-9a18-f1bef45050c0" (UID: "3fe1664f-f4d0-44c3-9a18-f1bef45050c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:11.049059 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.049023    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:11.049059 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.049041    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:11.050856 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.050833    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-dshm" (OuterVolumeSpecName: "dshm") pod "3fe1664f-f4d0-44c3-9a18-f1bef45050c0" (UID: "3fe1664f-f4d0-44c3-9a18-f1bef45050c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:11.050940 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.050884    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-kube-api-access-zlssj" (OuterVolumeSpecName: "kube-api-access-zlssj") pod "3fe1664f-f4d0-44c3-9a18-f1bef45050c0" (UID: "3fe1664f-f4d0-44c3-9a18-f1bef45050c0"). InnerVolumeSpecName "kube-api-access-zlssj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:43:11.050940 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.050918    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fe1664f-f4d0-44c3-9a18-f1bef45050c0" (UID: "3fe1664f-f4d0-44c3-9a18-f1bef45050c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:43:11.149618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.149578    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:11.149618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.149611    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlssj\" (UniqueName: \"kubernetes.io/projected/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-kube-api-access-zlssj\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:11.149618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.149620    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:11.149618 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.149630    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe1664f-f4d0-44c3-9a18-f1bef45050c0-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:11.309131 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.309090    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"]
Sep 17 14:43:11.313030 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.312999    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/sub-test-model-cc3801-kserve-58769c6fd7-647d8"]
Sep 17 14:43:11.932930 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:11.932876    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt" podUID="5d09a334-e389-47b0-901f-a43ad3f8836d" containerName="main" containerID="cri-o://f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b" gracePeriod=45
Sep 17 14:43:12.180609 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.180583    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:43:12.260693 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.260664    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d09a334-e389-47b0-901f-a43ad3f8836d-tls-certs\") pod \"5d09a334-e389-47b0-901f-a43ad3f8836d\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") "
Sep 17 14:43:12.260880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.260714    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cphv\" (UniqueName: \"kubernetes.io/projected/5d09a334-e389-47b0-901f-a43ad3f8836d-kube-api-access-6cphv\") pod \"5d09a334-e389-47b0-901f-a43ad3f8836d\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") "
Sep 17 14:43:12.260880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.260740    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-home\") pod \"5d09a334-e389-47b0-901f-a43ad3f8836d\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") "
Sep 17 14:43:12.260880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.260811    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-dshm\") pod \"5d09a334-e389-47b0-901f-a43ad3f8836d\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") "
Sep 17 14:43:12.260880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.260840    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-model-cache\") pod \"5d09a334-e389-47b0-901f-a43ad3f8836d\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") "
Sep 17 14:43:12.260880 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.260872    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-tmp-dir\") pod \"5d09a334-e389-47b0-901f-a43ad3f8836d\" (UID: \"5d09a334-e389-47b0-901f-a43ad3f8836d\") "
Sep 17 14:43:12.261152 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.261050    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-home" (OuterVolumeSpecName: "home") pod "5d09a334-e389-47b0-901f-a43ad3f8836d" (UID: "5d09a334-e389-47b0-901f-a43ad3f8836d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:12.261211 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.261148    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d09a334-e389-47b0-901f-a43ad3f8836d" (UID: "5d09a334-e389-47b0-901f-a43ad3f8836d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:12.261211 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.261196    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d09a334-e389-47b0-901f-a43ad3f8836d" (UID: "5d09a334-e389-47b0-901f-a43ad3f8836d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:12.261211 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.261174    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:12.262963 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.262938    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-dshm" (OuterVolumeSpecName: "dshm") pod "5d09a334-e389-47b0-901f-a43ad3f8836d" (UID: "5d09a334-e389-47b0-901f-a43ad3f8836d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:12.263330 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.263308    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d09a334-e389-47b0-901f-a43ad3f8836d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d09a334-e389-47b0-901f-a43ad3f8836d" (UID: "5d09a334-e389-47b0-901f-a43ad3f8836d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:43:12.263399 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.263323    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d09a334-e389-47b0-901f-a43ad3f8836d-kube-api-access-6cphv" (OuterVolumeSpecName: "kube-api-access-6cphv") pod "5d09a334-e389-47b0-901f-a43ad3f8836d" (UID: "5d09a334-e389-47b0-901f-a43ad3f8836d"). InnerVolumeSpecName "kube-api-access-6cphv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:43:12.361954 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.361903    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d09a334-e389-47b0-901f-a43ad3f8836d-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:12.361954 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.361943    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cphv\" (UniqueName: \"kubernetes.io/projected/5d09a334-e389-47b0-901f-a43ad3f8836d-kube-api-access-6cphv\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:12.361954 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.361956    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:12.362195 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.361969    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:12.362195 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.361981    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d09a334-e389-47b0-901f-a43ad3f8836d-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:43:12.965583 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:12.965552    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe1664f-f4d0-44c3-9a18-f1bef45050c0" path="/var/lib/kubelet/pods/3fe1664f-f4d0-44c3-9a18-f1bef45050c0/volumes"
Sep 17 14:43:13.000539 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:13.000503    2560 generic.go:356] "Generic (PLEG): container finished" podID="5d09a334-e389-47b0-901f-a43ad3f8836d" containerID="f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b" exitCode=0
Sep 17 14:43:13.000740 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:13.000548    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt" event={"ID":"5d09a334-e389-47b0-901f-a43ad3f8836d","Type":"ContainerDied","Data":"f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b"}
Sep 17 14:43:13.000740 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:13.000557    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"
Sep 17 14:43:13.000740 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:13.000586    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt" event={"ID":"5d09a334-e389-47b0-901f-a43ad3f8836d","Type":"ContainerDied","Data":"af386939e890cbbab7b429e820d240c0ea26d56b1b9c15b4ba676206766133aa"}
Sep 17 14:43:13.000740 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:13.000606    2560 scope.go:117] "RemoveContainer" containerID="f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b"
Sep 17 14:43:13.009489 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:13.009460    2560 scope.go:117] "RemoveContainer" containerID="f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b"
Sep 17 14:43:13.009786 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:43:13.009761    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b\": container with ID starting with f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b not found: ID does not exist" containerID="f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b"
Sep 17 14:43:13.009853 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:13.009796    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b"} err="failed to get container status \"f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b\": rpc error: code = NotFound desc = could not find container \"f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b\": container with ID starting with f1db7589618f0bcdefaaeb3b9ed980ce1b63e7b4cfdb69755c3aa52a7438da9b not found: ID does not exist"
Sep 17 14:43:13.019155 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:13.019127    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"]
Sep 17 14:43:13.025268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:13.025246    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc3801/rate-test-model-cc3801-kserve-79f988d547-nr2wt"]
Sep 17 14:43:14.965859 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:43:14.965827    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d09a334-e389-47b0-901f-a43ad3f8836d" path="/var/lib/kubelet/pods/5d09a334-e389-47b0-901f-a43ad3f8836d/volumes"
Sep 17 14:44:03.907065 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:03.907025    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-slhtt"]
Sep 17 14:44:03.907485 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:03.907225    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt" podUID="b391ea6c-16e1-4913-bae6-c4944296a762" containerName="manager" containerID="cri-o://ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504" gracePeriod=30
Sep 17 14:44:04.150238 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.150212    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:44:04.227969 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.227877    2560 generic.go:356] "Generic (PLEG): container finished" podID="b391ea6c-16e1-4913-bae6-c4944296a762" containerID="ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504" exitCode=0
Sep 17 14:44:04.227969 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.227919    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt"
Sep 17 14:44:04.227969 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.227947    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt" event={"ID":"b391ea6c-16e1-4913-bae6-c4944296a762","Type":"ContainerDied","Data":"ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504"}
Sep 17 14:44:04.228195 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.228007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6bd7d7cd-slhtt" event={"ID":"b391ea6c-16e1-4913-bae6-c4944296a762","Type":"ContainerDied","Data":"aed1fcc5e1b7940f8edc7ed9249b93441c13ac1f5b5bbf0ba77f4f54b5e28a73"}
Sep 17 14:44:04.228195 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.228036    2560 scope.go:117] "RemoveContainer" containerID="ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504"
Sep 17 14:44:04.238145 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.238124    2560 scope.go:117] "RemoveContainer" containerID="ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504"
Sep 17 14:44:04.238456 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:44:04.238437    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504\": container with ID starting with ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504 not found: ID does not exist" containerID="ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504"
Sep 17 14:44:04.238502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.238466    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504"} err="failed to get container status \"ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504\": rpc error: code = NotFound desc = could not find container \"ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504\": container with ID starting with ddeaac04ddd3490cc1fdffc6e0169d6bc57d285560f43d098549b53731273504 not found: ID does not exist"
Sep 17 14:44:04.251773 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.251751    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6wdc\" (UniqueName: \"kubernetes.io/projected/b391ea6c-16e1-4913-bae6-c4944296a762-kube-api-access-m6wdc\") pod \"b391ea6c-16e1-4913-bae6-c4944296a762\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") "
Sep 17 14:44:04.251869 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.251787    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-webhook-cert\") pod \"b391ea6c-16e1-4913-bae6-c4944296a762\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") "
Sep 17 14:44:04.251934 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.251915    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-metrics-certs\") pod \"b391ea6c-16e1-4913-bae6-c4944296a762\" (UID: \"b391ea6c-16e1-4913-bae6-c4944296a762\") "
Sep 17 14:44:04.253839 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.253813    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b391ea6c-16e1-4913-bae6-c4944296a762-kube-api-access-m6wdc" (OuterVolumeSpecName: "kube-api-access-m6wdc") pod "b391ea6c-16e1-4913-bae6-c4944296a762" (UID: "b391ea6c-16e1-4913-bae6-c4944296a762"). InnerVolumeSpecName "kube-api-access-m6wdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:04.253939 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.253855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b391ea6c-16e1-4913-bae6-c4944296a762" (UID: "b391ea6c-16e1-4913-bae6-c4944296a762"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:44:04.253939 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.253889    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b391ea6c-16e1-4913-bae6-c4944296a762" (UID: "b391ea6c-16e1-4913-bae6-c4944296a762"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:44:04.352945 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.352904    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:44:04.352945 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.352937    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6wdc\" (UniqueName: \"kubernetes.io/projected/b391ea6c-16e1-4913-bae6-c4944296a762-kube-api-access-m6wdc\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:44:04.352945 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.352946    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b391ea6c-16e1-4913-bae6-c4944296a762-webhook-cert\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:44:04.549832 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.549798    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-slhtt"]
Sep 17 14:44:04.554999 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.554970    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-slhtt"]
Sep 17 14:44:04.964352 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:44:04.964274    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b391ea6c-16e1-4913-bae6-c4944296a762" path="/var/lib/kubelet/pods/b391ea6c-16e1-4913-bae6-c4944296a762/volumes"
Sep 17 14:45:05.397529 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.397488    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-4p6rx"]
Sep 17 14:45:05.397931 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.397848    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe1664f-f4d0-44c3-9a18-f1bef45050c0" containerName="main"
Sep 17 14:45:05.397931 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.397859    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe1664f-f4d0-44c3-9a18-f1bef45050c0" containerName="main"
Sep 17 14:45:05.397931 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.397872    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b391ea6c-16e1-4913-bae6-c4944296a762" containerName="manager"
Sep 17 14:45:05.397931 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.397877    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b391ea6c-16e1-4913-bae6-c4944296a762" containerName="manager"
Sep 17 14:45:05.397931 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.397893    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d09a334-e389-47b0-901f-a43ad3f8836d" containerName="main"
Sep 17 14:45:05.397931 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.397899    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d09a334-e389-47b0-901f-a43ad3f8836d" containerName="main"
Sep 17 14:45:05.398117 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.397948    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b391ea6c-16e1-4913-bae6-c4944296a762" containerName="manager"
Sep 17 14:45:05.398117 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.397957    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d09a334-e389-47b0-901f-a43ad3f8836d" containerName="main"
Sep 17 14:45:05.398117 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.397964    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe1664f-f4d0-44c3-9a18-f1bef45050c0" containerName="main"
Sep 17 14:45:05.402157 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.402139    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:05.405888 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.405864    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 14:45:05.406030 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.405914    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 14:45:05.406030 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.405931    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p95mp\""
Sep 17 14:45:05.406030 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.405967    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 14:45:05.413721 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.413695    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-4p6rx"]
Sep 17 14:45:05.525773 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.525729    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-webhook-cert\") pod \"maas-controller-6b6bd7d7cd-4p6rx\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") " pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:05.525773 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.525767    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmqb6\" (UniqueName: \"kubernetes.io/projected/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-kube-api-access-lmqb6\") pod \"maas-controller-6b6bd7d7cd-4p6rx\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") " pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:05.526013 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.525804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-metrics-certs\") pod \"maas-controller-6b6bd7d7cd-4p6rx\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") " pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:05.626617 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.626562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-webhook-cert\") pod \"maas-controller-6b6bd7d7cd-4p6rx\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") " pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:05.626617 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.626618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmqb6\" (UniqueName: \"kubernetes.io/projected/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-kube-api-access-lmqb6\") pod \"maas-controller-6b6bd7d7cd-4p6rx\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") " pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:05.626883 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.626746    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-metrics-certs\") pod \"maas-controller-6b6bd7d7cd-4p6rx\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") " pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:05.629029 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.629009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-metrics-certs\") pod \"maas-controller-6b6bd7d7cd-4p6rx\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") " pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:05.629153 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.629046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-webhook-cert\") pod \"maas-controller-6b6bd7d7cd-4p6rx\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") " pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:05.636520 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.636493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmqb6\" (UniqueName: \"kubernetes.io/projected/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-kube-api-access-lmqb6\") pod \"maas-controller-6b6bd7d7cd-4p6rx\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") " pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:05.711937 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:05.711853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:06.002268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:06.002229    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-4p6rx"]
Sep 17 14:45:06.004615 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:45:06.004584    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c0c6a9_b6fd_4f33_888c_a4b1cca3dc66.slice/crio-fd0f915ad9ca71ea08f232219ea78a70179b2eea098a99363141189085600324 WatchSource:0}: Error finding container fd0f915ad9ca71ea08f232219ea78a70179b2eea098a99363141189085600324: Status 404 returned error can't find the container with id fd0f915ad9ca71ea08f232219ea78a70179b2eea098a99363141189085600324
Sep 17 14:45:06.005943 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:06.005927    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:45:06.509160 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:06.509129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx" event={"ID":"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66","Type":"ContainerStarted","Data":"fd0f915ad9ca71ea08f232219ea78a70179b2eea098a99363141189085600324"}
Sep 17 14:45:07.514702 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:07.514665    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx" event={"ID":"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66","Type":"ContainerStarted","Data":"93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91"}
Sep 17 14:45:07.515127 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:07.514808    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:07.535205 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:07.535157    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx" podStartSLOduration=1.966626465 podStartE2EDuration="2.535142068s" totalImagesPullingTime="568.515603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:45:07.533995091 +0000 UTC m=+2405.125170853" watchObservedRunningTime="2026-09-17 14:45:07.535142068 +0000 UTC m=+2405.126317854"
Sep 17 14:45:18.522277 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:18.522242    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:27.861334 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:27.861242    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-4p6rx"]
Sep 17 14:45:27.861912 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:27.861480    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx" podUID="35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66" containerName="manager" containerID="cri-o://93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91" gracePeriod=30
Sep 17 14:45:28.108426 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.108393    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:28.133780 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.133701    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmqb6\" (UniqueName: \"kubernetes.io/projected/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-kube-api-access-lmqb6\") pod \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") "
Sep 17 14:45:28.133780 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.133761    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-webhook-cert\") pod \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") "
Sep 17 14:45:28.134007 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.133803    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-metrics-certs\") pod \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\" (UID: \"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66\") "
Sep 17 14:45:28.136350 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.136288    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-kube-api-access-lmqb6" (OuterVolumeSpecName: "kube-api-access-lmqb6") pod "35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66" (UID: "35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66"). InnerVolumeSpecName "kube-api-access-lmqb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:45:28.136549 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.136452    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66" (UID: "35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:45:28.136738 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.136710    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66" (UID: "35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:45:28.234943 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.234898    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmqb6\" (UniqueName: \"kubernetes.io/projected/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-kube-api-access-lmqb6\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:45:28.234943 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.234934    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-webhook-cert\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:45:28.234943 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.234944    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:45:28.610503 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.610466    2560 generic.go:356] "Generic (PLEG): container finished" podID="35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66" containerID="93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91" exitCode=0
Sep 17 14:45:28.610672 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.610515    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx"
Sep 17 14:45:28.610672 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.610553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx" event={"ID":"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66","Type":"ContainerDied","Data":"93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91"}
Sep 17 14:45:28.610672 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.610611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6bd7d7cd-4p6rx" event={"ID":"35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66","Type":"ContainerDied","Data":"fd0f915ad9ca71ea08f232219ea78a70179b2eea098a99363141189085600324"}
Sep 17 14:45:28.610672 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.610635    2560 scope.go:117] "RemoveContainer" containerID="93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91"
Sep 17 14:45:28.619882 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.619864    2560 scope.go:117] "RemoveContainer" containerID="93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91"
Sep 17 14:45:28.620200 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:45:28.620182    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91\": container with ID starting with 93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91 not found: ID does not exist" containerID="93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91"
Sep 17 14:45:28.620248 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.620209    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91"} err="failed to get container status \"93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91\": rpc error: code = NotFound desc = could not find container \"93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91\": container with ID starting with 93d76b39d4c179e6c42ee884b11ed1e544348c739256aa06674318f439f94c91 not found: ID does not exist"
Sep 17 14:45:28.641863 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.641828    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-4p6rx"]
Sep 17 14:45:28.648408 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.648378    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b6bd7d7cd-4p6rx"]
Sep 17 14:45:28.966149 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:45:28.966064    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66" path="/var/lib/kubelet/pods/35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66/volumes"
Sep 17 14:48:52.801342 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.801260    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"]
Sep 17 14:48:52.801831 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.801619    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66" containerName="manager"
Sep 17 14:48:52.801831 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.801632    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66" containerName="manager"
Sep 17 14:48:52.801831 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.801713    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="35c0c6a9-b6fd-4f33-888c-a4b1cca3dc66" containerName="manager"
Sep 17 14:48:52.804678 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.804658    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.807441 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.807411    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-38ab7a6e-kserve-self-signed-certs\""
Sep 17 14:48:52.817653 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.817623    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-dshm\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.817800 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.817663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pntf\" (UniqueName: \"kubernetes.io/projected/5f5410e9-5a46-496c-a9e3-bf3413feec7e-kube-api-access-6pntf\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.817800 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.817695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-model-cache\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.817800 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.817768    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tmp-dir\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.817927 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.817860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-home\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.817927 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.817906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tls-certs\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.818143 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.818124    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"]
Sep 17 14:48:52.918979 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.918933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-dshm\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.918979 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.918977    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pntf\" (UniqueName: \"kubernetes.io/projected/5f5410e9-5a46-496c-a9e3-bf3413feec7e-kube-api-access-6pntf\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.919242 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.918997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-model-cache\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.919242 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.919016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tmp-dir\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.919242 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.919057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-home\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.919242 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.919085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tls-certs\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.919513 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.919484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-model-cache\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.919569 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.919491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tmp-dir\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.919569 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.919553    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-home\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.921226 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.921209    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-dshm\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.921503 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.921487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tls-certs\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:52.927855 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:52.927834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pntf\" (UniqueName: \"kubernetes.io/projected/5f5410e9-5a46-496c-a9e3-bf3413feec7e-kube-api-access-6pntf\") pod \"e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") " pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:53.114014 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:53.113924    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:48:53.396856 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:53.396769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"]
Sep 17 14:48:53.399905 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:48:53.399874    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f5410e9_5a46_496c_a9e3_bf3413feec7e.slice/crio-6a876494c428a226563e0194cc364f70a12feb62545222e8345480f2ddfa89f4 WatchSource:0}: Error finding container 6a876494c428a226563e0194cc364f70a12feb62545222e8345480f2ddfa89f4: Status 404 returned error can't find the container with id 6a876494c428a226563e0194cc364f70a12feb62545222e8345480f2ddfa89f4
Sep 17 14:48:53.516517 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:53.516477    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf" event={"ID":"5f5410e9-5a46-496c-a9e3-bf3413feec7e","Type":"ContainerStarted","Data":"5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970"}
Sep 17 14:48:53.516694 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:53.516526    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf" event={"ID":"5f5410e9-5a46-496c-a9e3-bf3413feec7e","Type":"ContainerStarted","Data":"6a876494c428a226563e0194cc364f70a12feb62545222e8345480f2ddfa89f4"}
Sep 17 14:48:53.555769 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:48:53.555708    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf" podStartSLOduration=1.555694278 podStartE2EDuration="1.555694278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:48:53.554693022 +0000 UTC m=+2631.145868785" watchObservedRunningTime="2026-09-17 14:48:53.555694278 +0000 UTC m=+2631.146870039"
Sep 17 14:49:03.114442 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:03.114413    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:49:03.114820 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:03.114650    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:49:03.127064 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:03.127042    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:49:03.572242 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:03.572209    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:49:12.382159 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.382116    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"]
Sep 17 14:49:12.385612 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.385588    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.388435 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.388408    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-38ab7a6e-kserve-self-signed-certs\""
Sep 17 14:49:12.397207 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.397179    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"]
Sep 17 14:49:12.496057 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.496020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-model-cache\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.496057 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.496062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4c8f\" (UniqueName: \"kubernetes.io/projected/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-kube-api-access-g4c8f\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.496348 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.496084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tls-certs\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.496348 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.496106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tmp-dir\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.496348 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.496238    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-home\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.496348 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.496288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-dshm\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.597729 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.597697    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tls-certs\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.597890 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.597742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tmp-dir\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.597890 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.597838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-home\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.597890 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.597871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-dshm\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.598051 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.597925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-model-cache\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.598115 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.598081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4c8f\" (UniqueName: \"kubernetes.io/projected/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-kube-api-access-g4c8f\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.598224 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.598202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tmp-dir\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.598304 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.598250    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-home\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.598304 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.598277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-model-cache\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.600269 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.600245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tls-certs\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.600530 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.600510    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-dshm\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.606449 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.606428    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4c8f\" (UniqueName: \"kubernetes.io/projected/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-kube-api-access-g4c8f\") pod \"e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") " pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.695063 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.694966    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:12.968806 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:12.968780    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"]
Sep 17 14:49:12.971482 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:49:12.971453    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d5133dc_6e40_44ef_b1ff_d652ad7ecdd2.slice/crio-7b172ca39afd72722175665ede9a3f51131b7ed9bf87cd1300644ed5ce4c26da WatchSource:0}: Error finding container 7b172ca39afd72722175665ede9a3f51131b7ed9bf87cd1300644ed5ce4c26da: Status 404 returned error can't find the container with id 7b172ca39afd72722175665ede9a3f51131b7ed9bf87cd1300644ed5ce4c26da
Sep 17 14:49:13.611106 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:13.611067    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds" event={"ID":"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2","Type":"ContainerStarted","Data":"fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792"}
Sep 17 14:49:13.611582 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:13.611112    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds" event={"ID":"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2","Type":"ContainerStarted","Data":"7b172ca39afd72722175665ede9a3f51131b7ed9bf87cd1300644ed5ce4c26da"}
Sep 17 14:49:13.631867 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:13.631820    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds" podStartSLOduration=1.6318054640000001 podStartE2EDuration="1.631805464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:49:13.630830932 +0000 UTC m=+2651.222006705" watchObservedRunningTime="2026-09-17 14:49:13.631805464 +0000 UTC m=+2651.222981225"
Sep 17 14:49:22.695185 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:22.695145    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:22.695669 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:22.695197    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:22.707418 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:22.707393    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:23.667110 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:23.667067    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:49:45.071614 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:45.071576    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"]
Sep 17 14:49:46.040774 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:49:46.040737    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"]
Sep 17 14:50:00.090427 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.090311    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf" podUID="5f5410e9-5a46-496c-a9e3-bf3413feec7e" containerName="main" containerID="cri-o://5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970" gracePeriod=45
Sep 17 14:50:00.340397 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.340353    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:50:00.458861 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.458769    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-home\") pod \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") "
Sep 17 14:50:00.458861 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.458837    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tmp-dir\") pod \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") "
Sep 17 14:50:00.459121 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.458869    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-model-cache\") pod \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") "
Sep 17 14:50:00.459121 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.458902    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pntf\" (UniqueName: \"kubernetes.io/projected/5f5410e9-5a46-496c-a9e3-bf3413feec7e-kube-api-access-6pntf\") pod \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") "
Sep 17 14:50:00.459121 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.458950    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-dshm\") pod \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") "
Sep 17 14:50:00.459121 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.458970    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tls-certs\") pod \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\" (UID: \"5f5410e9-5a46-496c-a9e3-bf3413feec7e\") "
Sep 17 14:50:00.459121 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.459091    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-home" (OuterVolumeSpecName: "home") pod "5f5410e9-5a46-496c-a9e3-bf3413feec7e" (UID: "5f5410e9-5a46-496c-a9e3-bf3413feec7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:00.459401 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.459123    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f5410e9-5a46-496c-a9e3-bf3413feec7e" (UID: "5f5410e9-5a46-496c-a9e3-bf3413feec7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:00.459401 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.459150    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f5410e9-5a46-496c-a9e3-bf3413feec7e" (UID: "5f5410e9-5a46-496c-a9e3-bf3413feec7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:00.459401 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.459281    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:00.459401 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.459299    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:00.459401 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.459313    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:00.461303 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.461271    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-dshm" (OuterVolumeSpecName: "dshm") pod "5f5410e9-5a46-496c-a9e3-bf3413feec7e" (UID: "5f5410e9-5a46-496c-a9e3-bf3413feec7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:00.461436 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.461317    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f5410e9-5a46-496c-a9e3-bf3413feec7e" (UID: "5f5410e9-5a46-496c-a9e3-bf3413feec7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:50:00.461436 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.461325    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5410e9-5a46-496c-a9e3-bf3413feec7e-kube-api-access-6pntf" (OuterVolumeSpecName: "kube-api-access-6pntf") pod "5f5410e9-5a46-496c-a9e3-bf3413feec7e" (UID: "5f5410e9-5a46-496c-a9e3-bf3413feec7e"). InnerVolumeSpecName "kube-api-access-6pntf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:50:00.560298 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.560255    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pntf\" (UniqueName: \"kubernetes.io/projected/5f5410e9-5a46-496c-a9e3-bf3413feec7e-kube-api-access-6pntf\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:00.560298 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.560298    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f5410e9-5a46-496c-a9e3-bf3413feec7e-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:00.560541 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.560312    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5410e9-5a46-496c-a9e3-bf3413feec7e-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:00.824643 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.824608    2560 generic.go:356] "Generic (PLEG): container finished" podID="5f5410e9-5a46-496c-a9e3-bf3413feec7e" containerID="5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970" exitCode=0
Sep 17 14:50:00.824843 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.824654    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"
Sep 17 14:50:00.824843 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.824664    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf" event={"ID":"5f5410e9-5a46-496c-a9e3-bf3413feec7e","Type":"ContainerDied","Data":"5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970"}
Sep 17 14:50:00.824843 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.824703    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf" event={"ID":"5f5410e9-5a46-496c-a9e3-bf3413feec7e","Type":"ContainerDied","Data":"6a876494c428a226563e0194cc364f70a12feb62545222e8345480f2ddfa89f4"}
Sep 17 14:50:00.824843 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.824725    2560 scope.go:117] "RemoveContainer" containerID="5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970"
Sep 17 14:50:00.833573 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.833550    2560 scope.go:117] "RemoveContainer" containerID="5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970"
Sep 17 14:50:00.833921 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:50:00.833899    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970\": container with ID starting with 5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970 not found: ID does not exist" containerID="5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970"
Sep 17 14:50:00.834024 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.833930    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970"} err="failed to get container status \"5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970\": rpc error: code = NotFound desc = could not find container \"5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970\": container with ID starting with 5f393da315522d36a4486c0b3cfee6fadfc9a19c8bd10329fa9327a720cd2970 not found: ID does not exist"
Sep 17 14:50:00.845168 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.845132    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"]
Sep 17 14:50:00.849273 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.849245    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-38ab7a6e-kserve-6b57df6b67-pdxjf"]
Sep 17 14:50:00.971481 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:00.971446    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f5410e9-5a46-496c-a9e3-bf3413feec7e" path="/var/lib/kubelet/pods/5f5410e9-5a46-496c-a9e3-bf3413feec7e/volumes"
Sep 17 14:50:01.061575 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.061531    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds" podUID="6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" containerName="main" containerID="cri-o://fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792" gracePeriod=45
Sep 17 14:50:01.310859 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.310835    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:50:01.472708 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.472618    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4c8f\" (UniqueName: \"kubernetes.io/projected/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-kube-api-access-g4c8f\") pod \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") "
Sep 17 14:50:01.472708 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.472707    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tls-certs\") pod \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") "
Sep 17 14:50:01.472910 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.472745    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-home\") pod \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") "
Sep 17 14:50:01.472910 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.472804    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tmp-dir\") pod \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") "
Sep 17 14:50:01.472910 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.472822    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-dshm\") pod \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") "
Sep 17 14:50:01.472910 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.472867    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-model-cache\") pod \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\" (UID: \"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2\") "
Sep 17 14:50:01.473110 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.473086    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" (UID: "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:01.473174 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.473108    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-home" (OuterVolumeSpecName: "home") pod "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" (UID: "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:01.473238 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.473216    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tmp-dir\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:01.473292 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.473236    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-home\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:01.473292 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.473250    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" (UID: "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:01.474886 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.474861    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-dshm" (OuterVolumeSpecName: "dshm") pod "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" (UID: "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:01.475286 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.475261    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" (UID: "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:50:01.475286 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.475271    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-kube-api-access-g4c8f" (OuterVolumeSpecName: "kube-api-access-g4c8f") pod "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" (UID: "6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2"). InnerVolumeSpecName "kube-api-access-g4c8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:50:01.573967 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.573926    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-tls-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:01.573967 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.573957    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-dshm\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:01.573967 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.573966    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-model-cache\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:01.573967 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.573977    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4c8f\" (UniqueName: \"kubernetes.io/projected/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2-kube-api-access-g4c8f\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:01.830940 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.830904    2560 generic.go:356] "Generic (PLEG): container finished" podID="6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" containerID="fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792" exitCode=0
Sep 17 14:50:01.831120 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.830946    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"
Sep 17 14:50:01.831120 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.830993    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds" event={"ID":"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2","Type":"ContainerDied","Data":"fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792"}
Sep 17 14:50:01.831120 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.831041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds" event={"ID":"6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2","Type":"ContainerDied","Data":"7b172ca39afd72722175665ede9a3f51131b7ed9bf87cd1300644ed5ce4c26da"}
Sep 17 14:50:01.831120 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.831060    2560 scope.go:117] "RemoveContainer" containerID="fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792"
Sep 17 14:50:01.839857 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.839838    2560 scope.go:117] "RemoveContainer" containerID="fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792"
Sep 17 14:50:01.840135 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:50:01.840116    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792\": container with ID starting with fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792 not found: ID does not exist" containerID="fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792"
Sep 17 14:50:01.840190 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.840144    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792"} err="failed to get container status \"fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792\": rpc error: code = NotFound desc = could not find container \"fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792\": container with ID starting with fa30fc54d98f3ea8ccedd788404ffb92122ef36a9cc38abc20cf86a38e671792 not found: ID does not exist"
Sep 17 14:50:01.853814 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.853781    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"]
Sep 17 14:50:01.858728 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:01.858700    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-38ab7a6e-kserve-558f945d6c-zckds"]
Sep 17 14:50:02.965975 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:02.965937    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" path="/var/lib/kubelet/pods/6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2/volumes"
Sep 17 14:50:13.409488 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.409449    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"]
Sep 17 14:50:13.409895 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.409853    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5410e9-5a46-496c-a9e3-bf3413feec7e" containerName="main"
Sep 17 14:50:13.409895 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.409865    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5410e9-5a46-496c-a9e3-bf3413feec7e" containerName="main"
Sep 17 14:50:13.409895 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.409873    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" containerName="main"
Sep 17 14:50:13.409895 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.409878    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" containerName="main"
Sep 17 14:50:13.410026 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.409939    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5410e9-5a46-496c-a9e3-bf3413feec7e" containerName="main"
Sep 17 14:50:13.410026 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.409947    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d5133dc-6e40-44ef-b1ff-d652ad7ecdd2" containerName="main"
Sep 17 14:50:13.412830 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.412809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.425525 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.425497    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"]
Sep 17 14:50:13.471377 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.471311    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gr7z\" (UniqueName: \"kubernetes.io/projected/8c80057f-937a-45cd-ac0c-db6b35d50931-kube-api-access-2gr7z\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.471566 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.471389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c80057f-937a-45cd-ac0c-db6b35d50931-maas-api-tls\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.471566 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.471457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c80057f-937a-45cd-ac0c-db6b35d50931-metrics-certs\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.471566 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.471505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c80057f-937a-45cd-ac0c-db6b35d50931-openshift-service-ca\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.572335 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.572289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c80057f-937a-45cd-ac0c-db6b35d50931-openshift-service-ca\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.572547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.572393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gr7z\" (UniqueName: \"kubernetes.io/projected/8c80057f-937a-45cd-ac0c-db6b35d50931-kube-api-access-2gr7z\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.572547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.572439    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c80057f-937a-45cd-ac0c-db6b35d50931-maas-api-tls\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.572547 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.572511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c80057f-937a-45cd-ac0c-db6b35d50931-metrics-certs\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.572937 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.572898    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c80057f-937a-45cd-ac0c-db6b35d50931-openshift-service-ca\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.574951 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.574933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c80057f-937a-45cd-ac0c-db6b35d50931-maas-api-tls\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.575015 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.574968    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c80057f-937a-45cd-ac0c-db6b35d50931-metrics-certs\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.581502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.581480    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gr7z\" (UniqueName: \"kubernetes.io/projected/8c80057f-937a-45cd-ac0c-db6b35d50931-kube-api-access-2gr7z\") pod \"maas-api-776dd778bc-gg6xq\" (UID: \"8c80057f-937a-45cd-ac0c-db6b35d50931\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:13.726730 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:13.726637    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:14.000996 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:14.000959    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"]
Sep 17 14:50:14.004562 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:50:14.004533    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c80057f_937a_45cd_ac0c_db6b35d50931.slice/crio-4815b4bd1a7d16f6e2113920ad74b437033bbbb0ee0c9081b1f6467374744d23 WatchSource:0}: Error finding container 4815b4bd1a7d16f6e2113920ad74b437033bbbb0ee0c9081b1f6467374744d23: Status 404 returned error can't find the container with id 4815b4bd1a7d16f6e2113920ad74b437033bbbb0ee0c9081b1f6467374744d23
Sep 17 14:50:14.005909 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:14.005891    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:50:14.893068 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:14.893033    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq" event={"ID":"8c80057f-937a-45cd-ac0c-db6b35d50931","Type":"ContainerStarted","Data":"b9df2c0dbf970820daa91a346539d594980c0758dddd7d51fe3ba07717609de7"}
Sep 17 14:50:14.893500 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:14.893084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq" event={"ID":"8c80057f-937a-45cd-ac0c-db6b35d50931","Type":"ContainerStarted","Data":"4815b4bd1a7d16f6e2113920ad74b437033bbbb0ee0c9081b1f6467374744d23"}
Sep 17 14:50:14.893500 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:14.893129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:14.914538 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:14.914484    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq" podStartSLOduration=1.613591552 podStartE2EDuration="1.914466555s" totalImagesPullingTime="300.875003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:50:14.913990729 +0000 UTC m=+2712.505166506" watchObservedRunningTime="2026-09-17 14:50:14.914466555 +0000 UTC m=+2712.505642317"
Sep 17 14:50:20.905748 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:20.905717    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-776dd778bc-gg6xq"
Sep 17 14:50:20.966433 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:20.966397    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"]
Sep 17 14:50:20.966690 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:20.966654    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg" podUID="b160c3cc-fc7b-464f-9760-5c5d299f62fb" containerName="maas-api" containerID="cri-o://b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5" gracePeriod=30
Sep 17 14:50:21.688447 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:21.688399    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg" podUID="b160c3cc-fc7b-464f-9760-5c5d299f62fb" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.28:8443/health\": dial tcp 10.133.0.28:8443: connect: connection refused"
Sep 17 14:50:22.211179 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.211147    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:50:22.250527 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.250489    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgx4k\" (UniqueName: \"kubernetes.io/projected/b160c3cc-fc7b-464f-9760-5c5d299f62fb-kube-api-access-sgx4k\") pod \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") "
Sep 17 14:50:22.250527 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.250535    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-metrics-certs\") pod \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") "
Sep 17 14:50:22.250806 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.250597    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b160c3cc-fc7b-464f-9760-5c5d299f62fb-openshift-service-ca\") pod \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") "
Sep 17 14:50:22.250806 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.250663    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-maas-api-tls\") pod \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\" (UID: \"b160c3cc-fc7b-464f-9760-5c5d299f62fb\") "
Sep 17 14:50:22.251042 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.251016    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b160c3cc-fc7b-464f-9760-5c5d299f62fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b160c3cc-fc7b-464f-9760-5c5d299f62fb" (UID: "b160c3cc-fc7b-464f-9760-5c5d299f62fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:50:22.252836 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.252799    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b160c3cc-fc7b-464f-9760-5c5d299f62fb" (UID: "b160c3cc-fc7b-464f-9760-5c5d299f62fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:50:22.252948 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.252882    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b160c3cc-fc7b-464f-9760-5c5d299f62fb" (UID: "b160c3cc-fc7b-464f-9760-5c5d299f62fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:50:22.252948 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.252895    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b160c3cc-fc7b-464f-9760-5c5d299f62fb-kube-api-access-sgx4k" (OuterVolumeSpecName: "kube-api-access-sgx4k") pod "b160c3cc-fc7b-464f-9760-5c5d299f62fb" (UID: "b160c3cc-fc7b-464f-9760-5c5d299f62fb"). InnerVolumeSpecName "kube-api-access-sgx4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:50:22.351755 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.351663    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-maas-api-tls\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:22.351755 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.351695    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgx4k\" (UniqueName: \"kubernetes.io/projected/b160c3cc-fc7b-464f-9760-5c5d299f62fb-kube-api-access-sgx4k\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:22.351755 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.351706    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b160c3cc-fc7b-464f-9760-5c5d299f62fb-metrics-certs\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:22.351755 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.351716    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b160c3cc-fc7b-464f-9760-5c5d299f62fb-openshift-service-ca\") on node \"ip-10-0-131-146.ec2.internal\" DevicePath \"\""
Sep 17 14:50:22.931460 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.931423    2560 generic.go:356] "Generic (PLEG): container finished" podID="b160c3cc-fc7b-464f-9760-5c5d299f62fb" containerID="b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5" exitCode=0
Sep 17 14:50:22.931644 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.931466    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"
Sep 17 14:50:22.931644 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.931488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg" event={"ID":"b160c3cc-fc7b-464f-9760-5c5d299f62fb","Type":"ContainerDied","Data":"b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5"}
Sep 17 14:50:22.931644 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.931537    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg" event={"ID":"b160c3cc-fc7b-464f-9760-5c5d299f62fb","Type":"ContainerDied","Data":"6811794ca8af6aaf01023cf94d06c860a3d1330d99383210f496b90563965f4e"}
Sep 17 14:50:22.931644 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.931561    2560 scope.go:117] "RemoveContainer" containerID="b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5"
Sep 17 14:50:22.941230 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.941209    2560 scope.go:117] "RemoveContainer" containerID="b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5"
Sep 17 14:50:22.941540 ip-10-0-131-146 kubenswrapper[2560]: E0917 14:50:22.941519    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5\": container with ID starting with b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5 not found: ID does not exist" containerID="b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5"
Sep 17 14:50:22.941591 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.941550    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5"} err="failed to get container status \"b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5\": rpc error: code = NotFound desc = could not find container \"b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5\": container with ID starting with b0d4759f6a34bca7d4f5671453b4a830a13e106e71f801dd987dfb00cd6a9bf5 not found: ID does not exist"
Sep 17 14:50:22.954096 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.954070    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"]
Sep 17 14:50:22.964699 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:22.964671    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-f2bcg"]
Sep 17 14:50:24.965625 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:24.965577    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b160c3cc-fc7b-464f-9760-5c5d299f62fb" path="/var/lib/kubelet/pods/b160c3cc-fc7b-464f-9760-5c5d299f62fb/volumes"
Sep 17 14:50:43.328712 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:43.328684    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7895599765-pn78d_d9adba57-c979-487b-81d7-62107b241419/copy-manifests/0.log"
Sep 17 14:50:43.337433 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:43.337397    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7895599765-pn78d_d9adba57-c979-487b-81d7-62107b241419/manager/0.log"
Sep 17 14:50:43.808763 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:43.808728    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-776dd778bc-gg6xq_8c80057f-937a-45cd-ac0c-db6b35d50931/maas-api/0.log"
Sep 17 14:50:45.323505 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:45.323480    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-rgqch_a4804acc-0724-42db-8263-52f1611d5892/kuadrant-console-plugin/0.log"
Sep 17 14:50:46.301004 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:46.300958    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6769664b-6wdck_483f6869-a5b2-44e7-9cbc-77ccb3592e3a/kube-auth-proxy/0.log"
Sep 17 14:50:46.791268 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:46.791239    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zvgmc_17506b00-b587-4549-a48c-a11211e2a6bd/payload-processing/0.log"
Sep 17 14:50:47.275043 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:47.275009    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zdvcf_6f1228c3-5493-4572-baef-43101267ddaa/main/0.log"
Sep 17 14:50:47.415746 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:47.415721    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-s7w4m_927b3b41-530b-4681-8f99-fc403b39cc91/main/0.log"
Sep 17 14:50:47.777491 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:47.777464    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhcs72_a16b7929-363b-473f-aac0-351c848c1717/main/0.log"
Sep 17 14:50:48.012156 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:48.012124    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-x7jz7_6bfad0ac-2050-404d-939d-5cf8cbb8da7a/main/0.log"
Sep 17 14:50:56.128405 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:56.128376    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-52kgw_3839cd65-b24b-4e79-b075-d703429c694f/global-pull-secret-syncer/0.log"
Sep 17 14:50:56.390950 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:56.390875    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wr9bq_29b2af19-f2bf-42d3-9716-639cb0848d7b/konnectivity-agent/0.log"
Sep 17 14:50:56.417256 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:50:56.417222    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-146.ec2.internal_f2a8032d1e1ba5143915f9592a048d25/haproxy/0.log"
Sep 17 14:51:01.172407 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:01.172351    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-rgqch_a4804acc-0724-42db-8263-52f1611d5892/kuadrant-console-plugin/0.log"
Sep 17 14:51:02.709821 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:02.709791    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b610e21-46df-49f0-bc86-2cbd5691d2ee/alertmanager/0.log"
Sep 17 14:51:02.731166 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:02.731137    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b610e21-46df-49f0-bc86-2cbd5691d2ee/config-reloader/0.log"
Sep 17 14:51:02.752056 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:02.752026    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b610e21-46df-49f0-bc86-2cbd5691d2ee/kube-rbac-proxy-web/0.log"
Sep 17 14:51:02.775536 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:02.775510    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b610e21-46df-49f0-bc86-2cbd5691d2ee/kube-rbac-proxy/0.log"
Sep 17 14:51:02.795953 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:02.795921    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b610e21-46df-49f0-bc86-2cbd5691d2ee/kube-rbac-proxy-metric/0.log"
Sep 17 14:51:02.815871 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:02.815842    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b610e21-46df-49f0-bc86-2cbd5691d2ee/prom-label-proxy/0.log"
Sep 17 14:51:02.838902 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:02.838874    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b610e21-46df-49f0-bc86-2cbd5691d2ee/init-config-reloader/0.log"
Sep 17 14:51:03.174404 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.174377    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-txsnw_442f751d-a8a9-43d2-ae0c-edaaa0092a52/node-exporter/0.log"
Sep 17 14:51:03.192920 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.192886    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-txsnw_442f751d-a8a9-43d2-ae0c-edaaa0092a52/kube-rbac-proxy/0.log"
Sep 17 14:51:03.213642 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.213615    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-txsnw_442f751d-a8a9-43d2-ae0c-edaaa0092a52/init-textfile/0.log"
Sep 17 14:51:03.498689 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.498659    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f/prometheus/0.log"
Sep 17 14:51:03.518102 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.518073    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f/config-reloader/0.log"
Sep 17 14:51:03.540764 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.540734    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f/thanos-sidecar/0.log"
Sep 17 14:51:03.568502 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.568474    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f/kube-rbac-proxy-web/0.log"
Sep 17 14:51:03.589804 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.589777    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f/kube-rbac-proxy/0.log"
Sep 17 14:51:03.608662 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.608639    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f/kube-rbac-proxy-thanos/0.log"
Sep 17 14:51:03.628384 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.628342    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1c5fe8e4-e47b-4858-97b5-cee64e7fbc7f/init-config-reloader/0.log"
Sep 17 14:51:03.825778 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.825697    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fb5f4bb8-rhbgq_4d3faec5-4509-4524-921b-cffb4fcae95b/thanos-query/0.log"
Sep 17 14:51:03.852773 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.852742    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fb5f4bb8-rhbgq_4d3faec5-4509-4524-921b-cffb4fcae95b/kube-rbac-proxy-web/0.log"
Sep 17 14:51:03.874094 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.874070    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fb5f4bb8-rhbgq_4d3faec5-4509-4524-921b-cffb4fcae95b/kube-rbac-proxy/0.log"
Sep 17 14:51:03.894384 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.894341    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fb5f4bb8-rhbgq_4d3faec5-4509-4524-921b-cffb4fcae95b/prom-label-proxy/0.log"
Sep 17 14:51:03.914279 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.914250    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fb5f4bb8-rhbgq_4d3faec5-4509-4524-921b-cffb4fcae95b/kube-rbac-proxy-rules/0.log"
Sep 17 14:51:03.936015 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:03.935982    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fb5f4bb8-rhbgq_4d3faec5-4509-4524-921b-cffb4fcae95b/kube-rbac-proxy-metrics/0.log"
Sep 17 14:51:06.418623 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.418579    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"]
Sep 17 14:51:06.419012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.418956    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b160c3cc-fc7b-464f-9760-5c5d299f62fb" containerName="maas-api"
Sep 17 14:51:06.419012 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.418969    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b160c3cc-fc7b-464f-9760-5c5d299f62fb" containerName="maas-api"
Sep 17 14:51:06.419082 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.419041    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b160c3cc-fc7b-464f-9760-5c5d299f62fb" containerName="maas-api"
Sep 17 14:51:06.423652 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.423627    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.426377 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.426336    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-82mm4\"/\"kube-root-ca.crt\""
Sep 17 14:51:06.427338 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.427320    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-82mm4\"/\"default-dockercfg-qm296\""
Sep 17 14:51:06.427450 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.427377    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-82mm4\"/\"openshift-service-ca.crt\""
Sep 17 14:51:06.432626 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.432598    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"]
Sep 17 14:51:06.476123 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.476082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-sys\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.476123 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.476126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-proc\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.476413 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.476201    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-podres\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.476413 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.476269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5d58\" (UniqueName: \"kubernetes.io/projected/898cc110-0fe4-472e-ba28-dca076da8bbe-kube-api-access-d5d58\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.476413 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.476300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-lib-modules\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.577733 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.577691    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-podres\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.577913 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.577773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5d58\" (UniqueName: \"kubernetes.io/projected/898cc110-0fe4-472e-ba28-dca076da8bbe-kube-api-access-d5d58\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.577913 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.577811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-lib-modules\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.577913 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.577863    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-sys\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.577913 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.577896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-proc\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.578103 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.577865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-podres\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.578103 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.577934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-lib-modules\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.578103 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.577970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-sys\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.578103 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.577989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/898cc110-0fe4-472e-ba28-dca076da8bbe-proc\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.588988 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.588953    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5d58\" (UniqueName: \"kubernetes.io/projected/898cc110-0fe4-472e-ba28-dca076da8bbe-kube-api-access-d5d58\") pod \"perf-node-gather-daemonset-t5t7d\" (UID: \"898cc110-0fe4-472e-ba28-dca076da8bbe\") " pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:06.734423 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:06.734393    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:07.006734 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:07.006707    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"]
Sep 17 14:51:07.009347 ip-10-0-131-146 kubenswrapper[2560]: W0917 14:51:07.009316    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod898cc110_0fe4_472e_ba28_dca076da8bbe.slice/crio-7ea47f524eb79fcdbeb59db7df209e6d40c180c5b2a1499502ada2379596ef92 WatchSource:0}: Error finding container 7ea47f524eb79fcdbeb59db7df209e6d40c180c5b2a1499502ada2379596ef92: Status 404 returned error can't find the container with id 7ea47f524eb79fcdbeb59db7df209e6d40c180c5b2a1499502ada2379596ef92
Sep 17 14:51:07.129586 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:07.129549    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d" event={"ID":"898cc110-0fe4-472e-ba28-dca076da8bbe","Type":"ContainerStarted","Data":"7ea47f524eb79fcdbeb59db7df209e6d40c180c5b2a1499502ada2379596ef92"}
Sep 17 14:51:08.135195 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:08.135148    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d" event={"ID":"898cc110-0fe4-472e-ba28-dca076da8bbe","Type":"ContainerStarted","Data":"4f1647a8144a553a5a0f03bf998792fc9a55e46bf22b3396207be05066586487"}
Sep 17 14:51:08.135705 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:08.135230    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:08.154576 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:08.154512    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d" podStartSLOduration=2.154492258 podStartE2EDuration="2.154492258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:51:08.153627058 +0000 UTC m=+2765.744802821" watchObservedRunningTime="2026-09-17 14:51:08.154492258 +0000 UTC m=+2765.745668019"
Sep 17 14:51:08.546631 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:08.546592    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hm4vd_6c43836f-f444-4fa9-9155-b75f04b32d80/dns/0.log"
Sep 17 14:51:08.567905 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:08.567872    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hm4vd_6c43836f-f444-4fa9-9155-b75f04b32d80/kube-rbac-proxy/0.log"
Sep 17 14:51:08.671402 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:08.671372    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gc5s9_26394392-22b2-42a8-98b8-a6b6f194f569/dns-node-resolver/0.log"
Sep 17 14:51:09.300720 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:09.300674    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-544f7f87c9-dt7kb_f6690262-a99d-466e-bfbb-45d3e609d5ac/registry/0.log"
Sep 17 14:51:09.346349 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:09.346320    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h2l92_d18eac27-3f6f-4757-94c9-0e01cd87d96a/node-ca/0.log"
Sep 17 14:51:11.052337 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:11.052297    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6769664b-6wdck_483f6869-a5b2-44e7-9cbc-77ccb3592e3a/kube-auth-proxy/0.log"
Sep 17 14:51:11.188745 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:11.188711    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zvgmc_17506b00-b587-4549-a48c-a11211e2a6bd/payload-processing/0.log"
Sep 17 14:51:12.058512 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:12.058475    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-klkvh_823cdcd0-1f52-455f-98aa-1db092020587/serve-healthcheck-canary/0.log"
Sep 17 14:51:12.707551 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:12.707520    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g6ww4_40712e5d-89b9-4fcd-8561-769d4a59cd59/kube-rbac-proxy/0.log"
Sep 17 14:51:12.725946 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:12.725907    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g6ww4_40712e5d-89b9-4fcd-8561-769d4a59cd59/exporter/0.log"
Sep 17 14:51:12.744639 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:12.744600    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g6ww4_40712e5d-89b9-4fcd-8561-769d4a59cd59/extractor/0.log"
Sep 17 14:51:14.150518 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:14.150492    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-82mm4/perf-node-gather-daemonset-t5t7d"
Sep 17 14:51:15.654896 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:15.654818    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7895599765-pn78d_d9adba57-c979-487b-81d7-62107b241419/manager/0.log"
Sep 17 14:51:15.721210 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:15.721174    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7895599765-pn78d_d9adba57-c979-487b-81d7-62107b241419/copy-manifests/0.log"
Sep 17 14:51:17.150984 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:17.150948    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5cb7bb5755-54rzn_9c8ff066-59f3-4894-8707-190e212c8c01/manager/0.log"
Sep 17 14:51:18.062655 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:18.062566    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6769664b-6wdck_483f6869-a5b2-44e7-9cbc-77ccb3592e3a/kube-auth-proxy/0.log"
Sep 17 14:51:18.166069 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:18.166031    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zvgmc_17506b00-b587-4549-a48c-a11211e2a6bd/payload-processing/0.log"
Sep 17 14:51:20.374641 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:20.374610    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6769664b-6wdck_483f6869-a5b2-44e7-9cbc-77ccb3592e3a/kube-auth-proxy/0.log"
Sep 17 14:51:20.485057 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:20.485026    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zvgmc_17506b00-b587-4549-a48c-a11211e2a6bd/payload-processing/0.log"
Sep 17 14:51:23.650123 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:23.650093    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cns4_c1c3e777-dd90-4fea-9735-157ef5f2ebb7/kube-multus-additional-cni-plugins/0.log"
Sep 17 14:51:23.669189 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:23.669161    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cns4_c1c3e777-dd90-4fea-9735-157ef5f2ebb7/egress-router-binary-copy/0.log"
Sep 17 14:51:23.688518 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:23.688487    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cns4_c1c3e777-dd90-4fea-9735-157ef5f2ebb7/cni-plugins/0.log"
Sep 17 14:51:23.708311 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:23.708283    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cns4_c1c3e777-dd90-4fea-9735-157ef5f2ebb7/bond-cni-plugin/0.log"
Sep 17 14:51:23.728181 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:23.728150    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cns4_c1c3e777-dd90-4fea-9735-157ef5f2ebb7/routeoverride-cni/0.log"
Sep 17 14:51:23.746737 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:23.746694    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cns4_c1c3e777-dd90-4fea-9735-157ef5f2ebb7/whereabouts-cni-bincopy/0.log"
Sep 17 14:51:23.770176 ip-10-0-131-146 kubenswrapper[2560]: I0917 14:51:23.770147    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7cns4_c1c3e777-dd90-4fea-9735-157ef5f2ebb7/whereabouts-cni/0.log"